[AG-TECH] Problems uploading files

[Christoph Willing]

On 27/04/2010, at 2:32 PM, John I. Quebedeaux, Jr wrote:

> Hmmm... And ditto to Fedora 12, and 'ding' I may be having the same  
> problem.
> When I look for the files, they appear to be '0' sized though... But I
> haven't tried a 'small' one yet.


Thanks for adding to the evidence John.

It seems there are two aspects to this problem. On the server side,  
the file is not received resulting in a zero size file. However I  
don't anymore think this is a data size issue - it just seemed that  
way to me at first. The AG's FTPSServer prepares to start but no data  
at all is actually exchanged between the client and the server.

There is a client side problem too, that even when a file has uploaded  
correctly it may not appear in the venue client ui.

Its a very weird problem. I spent most of today with an "old" system  
(in which server & client sides both work perfectly) and gradually  
built & installed updated packages so that they all match a current  
non-working system - figuring that the fault would manifest after one  
of these updates, identifying it as the culprit. "Unfortunately" the  
system still works at the end of all that - it seems none of the prime  
suspect packages is causing the problem. Tomorrow I'll either have to  
come with a new strategy or new suspect support packages. Does anyone  
want to suspect, say, openssl or even python itself?


chris


>> From: Christoph Willing <c.willing at uq.edu.au>
>> Date: Tue, 27 Apr 2010 10:04:58 +1000
>> To: Jesus Cea Oliva <jesus.cea at uca.es>
>> Cc: <ag-tech at mcs.anl.gov>
>> Subject: Re: [AG-TECH] Problems uploading files
>>
>> Jesus,
>>
>> Could you submit a bug report for this problem please? You can do  
>> that
>> via the VenueClient Help menu.
>>
>>
>> I have now been able to replicate the problem on several systems -
>> Ubuntu, Slackware, Fedora & Debian. Interestingly older versions
>> running on these systems seem to work OK i.e. when a venue server  
>> runs
>> on the older systems, the problem is not evident. Unfortunately I
>> don't have resources to test all Linux systems but so far, the last
>> known versions to work correctly (a client can upload to a server
>> running on the same system) are:
>> Ubuntu Intrepid
>> Slackware 12.2
>> Fedora - unknown exactly but the problem is evident in Fedora 11
>> Debian - Lenny partially works
>>
>> The Debian case is interesting because a Lenny client will not upload
>> data to a Lenny server, yet a Jaunty client can upload to a Lenny
>> server (even though it can't upload to a Jaunty server).
>>
>>
>> At this stage, I can't determine whether the problem is due to AG
>> toolkit code or to one of the supporting software packages;  
>> particular
>> versions of zsi and m2crypto have caused most heartburn in the past -
>> maybe they're implicated again. I think it will take some time to  
>> find
>> and fix
>>
>>
>> chris
>>
>>
>> On 26/04/2010, at 12:51 AM, Jesus Cea Oliva wrote:
>>
>>> Hi Christoph, thanks for your indications :).
>>>
>>> I tried local VenueServers on machine, I mean, if I run Ubuntu, I
>>> execute in the same machine a local VenueServer, then, I run
>>> VenueClient and I try upload a file, the same with Windows 7.
>>>
>>> However, due to your mail, I've doing some tests:
>>>
>>> - On my Ubuntu system, I've entered at
>>> https://vv3.ap-accessgrid.org:8000/Venues/default
>>> , the Test Room, and I download files: VenueThermoClient.agpkg3 and
>>> methanol.obj, and rename it adding X (metahnol.objX). Then I can
>>> upload the files without problems to the server.
>>>
>>> I create a local network with 2 machines, with Windows 7 and Ubuntu.
>>>
>>> - Firstly I've executed a VenueServer on Ubuntu machine, then I've
>>> run VenueClients in both machines. I can't upload files in the 2
>>> machines (well you know, really, I can upload, but the Client
>>> appears stop, reentering in the Venue I can see the files uploaded).
>>> I've noticed other problem, when I try delete a file, VenueClient
>>> don't refresh this command, the system don't delete the file.
>>>
>>>
>>> - I've executed a VenueServer on Windows 7 machine, then I've run
>>> VenueClients in both machines. Here, I can upload files without
>>> problems, on Windows and Ubuntu systems.
>>>
>>> So, now, I know that I have problems with our VenueServer on Ubuntu
>>> systems, I don't know if I have to configure something or if there
>>> are problems about names, like ñ, spaces, etc (user name, path file,
>>> etc, although my home path is /home/ercea and, there, is the
>>> VenueServer's Data folder)
>>>
>>> Now I'll try change the system language to US and I'll repeat the
>>> tests.
>>>
>>>
>>> Thanks again and apologize for any inconveniences
>>> Regards!
>>>
>>>
>>>
>>> El dia 25 abr 2010 12:11, Christoph Willing <c.willing at uq.edu.au>
>>> escribió:
>>> On 23/04/2010, at 6:24 PM, Jesus Cea Oliva wrote:
>>>
>>> Hi Crhistoph, thanks for your response :)
>>>
>>> I try upload files since 1 o 2 K (source codes, txts, etc.) and
>>> files about 3-6 Mb (PDFs, ppts...) and, all cases I have the same
>>> problem. The strange of this problem is that ocurrs randomly, and
>>> sometimes, the same file that before appears not upload, now upload
>>> (although, the majority cases, don't upload)
>>>
>>> Are you using the same venue server in each test case (Ubuntu &
>>> Windows VenueClient)?
>>>
>>>
>>> Are you using your own server or a "well known" server?
>>> If these tests are with you own server, could you instead use the
>>> APAG venue server at https://vv3.ap-accessgrid.org:8000/Venues/
>>> default and then enter the Test venue. There are several files in
>>> that venue; could you try to download the simple.py and methanol.obj
>>> files from that venue and rename them e.g. simple.py -> simple.pyX
>>> and methanol.obj -> methanol.objX. Now upload te renamed files back
>>> into the Test venue. Do they both upload as expected?
>>>
>>>
>>> FYI, although I thought I had replicated the problem, I now find
>>> that it actually works correctly. My first hurried tests only
>>> appeared to replicate your symptoms - because I didn't wait long
>>> enogh for the upload to complete. However I have now performed the
>>> tests again and all files upload correctly - and the venue client
>>> interface shows the new data correctly in all cases.
>>>
>>> I believe the previous test appeared not to update the venue client
>>> interface because I was performing the tests from home using an ADSL
>>> connection - therefore the initial download was quite fast. The
>>> upload was slower because of the much lower upstream bandwidth of my
>>> ADSL connection. My guess is that the first portion of upload
>>> _appears_ fast because the venue client is just filling up the local
>>> TCP system buffer. However that buffer then has to be transferred
>>> (via slow ADSL uplink) to the venue server - which manifests as a
>>> fast intitial movement of the progress bar that then appears to
>>> freeze. The "freeze" is really just a pause while contents of the
>>> TCP buffer is being transferred across the network.
>>>
>>> In my case the simple.py file, being small, transfers very quickly.
>>> The methanol.obj file is about 11M - I guess this is just a bit
>>> larger than my system network buffer size - and takes a lot longer
>>> to transfer.
>>>
>>> Based on redoing the tests (using the APAG veneu server), I cannot
>>> replicate the problem.
>>>
>>> Could you confirm whether you still have the problem when you use
>>> the APAG venue server?
>>>
>>>
>>> chris
>>>
>>>
>>> On Windows I try similar files (somes files are the same), only
>>> don't work once due to a very big file. As I saw that I can upload
>>> files without problems... I try all file types. So I upload a file
>>> about 100Mb and don't work xD.
>>>
>>> Thanks for your help :). Regards!
>>>
>>> El dia 23 abr 2010 02:35, Christoph Willing <c.willing at uq.edu.au>
>>> escribió:
>>> I wonder if this is related to the size of the data object. I just
>>> tried (with Ubuntu 9.10 64bit) and found that a small file (6K)
>>> there was no problem. With a larger file (12M) I saw exactly the
>>> same problem - upload appears not to complete but has actually
>>> succeeded. Re-entering the venue shows that the new data is there.
>>>
>>> In your Windows test, are you uploading the same file (or something
>>> of similar size) as in the Ubuntu case?
>>>
>>> I will now be away for a few days so I can't do any more
>>> investigation myself until I return. Maybe others can look into this
>>> in the meantime ...
>>>
>>>
>>> chris
>>>
>>>
>>> On 23/04/2010, at 5:28 AM, Jesus Cea Oliva wrote:
>>>
>>> Thanks Tom for your fast response :).
>>>
>>> Well, atach the VenueClient log in this email, I can't see any
>>> problems :S.
>>>
>>>
>>>
>>> 04/22/10 21:16:50 140471913264880 VenueClient VenueClientUI.py: 1308
>>> DEBUG VenueClientUI.AddDataCB: Trying to upload to
>>> 'ftps://localhost:8006/7f00010133381d7afaba1c148fcabe1007'
>>> 04/22/10 21:17:00 140471913264880 VenueClient VenueClientUI.py: 1395
>>> DEBUG AddDataCB: URI of parent is
>>> 04/22/10 21:17:00 140471913264880 VenueClientController
>>> VenueClientController.py:821 DEBUG In
>>> VenueClientController.UploadVenueFiles
>>> 04/22/10 21:17:00 140471913264880 VenueClientController
>>> VenueClientController.py:822 DEBUG fileList = [u'/home/ercea/
>>> Descargas/DSD-Tutorial-3.pdf']
>>> 04/22/10 21:17:00 140471913264880 VenueClientController
>>> VenueClientController.py:848 DEBUG Serverpath:
>>> 04/22/10 21:17:00 140471913264880 VenueClientController
>>> VenueClientController.py:857 DEBUG Have args, creating thread, url:
>>> ftps://localhost:8006/7f00010133381d7afaba1c148fcabe1007
>>> , files: [u'/home/ercea/Descargas/DSD-Tutorial-3.pdf']
>>> 04/22/10 21:17:00 140471479552272 VenueClientController
>>> VenueClientController.py:944 DEBUG Upload: getting identity
>>> 04/22/10 21:17:00 140471479552272 VenueClientController
>>> VenueClientController.py:949 DEBUG Got identity <?xml
>>> version="1.0" ? ><Subject><Subject auth_data="" auth_type="x509"
>>> name="/O=Access Grid/O=Argonne National Laboratory/OU=Futures Lab
>>> Anonymous Authority/CN=Anonymous User
>>> 17130398eef9fd82cba8dc9d845a1768"/></ Subject>
>>> 04/22/10 21:17:00 140471479552272 VenueClientController
>>> VenueClientController.py:950 DEBUG get_ident_and_upload: Upload URL
>>> ftps://localhost:8006/7f00010133381d7afaba1c148fcabe1007
>>> 04/22/10 21:17:00 140471479552272 VenueClientController
>>> VenueClientController.py:953 DEBUG Get_ident_and_upload: Word is:
>>> ftps:
>>> 04/22/10 21:17:00 140471479552272 VenueClientController
>>> VenueClientController.py:953 DEBUG Get_ident_and_upload: Word is:
>>> 04/22/10 21:17:00 140471479552272 VenueClientController
>>> VenueClientController.py:953 DEBUG Get_ident_and_upload: Word is:
>>> localhost:8006
>>> 04/22/10 21:17:00 140471479552272 VenueClientController
>>> VenueClientController.py:953 DEBUG Get_ident_and_upload: Word is:
>>> 7f00010133381d7afaba1c148fcabe1007
>>> 04/22/10 21:17:00 140471479552272 DataStore DataStore.py:1267 INFO
>>> UploadFiles: ftps://localhost:
>>> 8006/7f00010133381d7afaba1c148fcabe1007 [u'/home/ercea/Descargas/
>>> DSD- Tutorial-3.pdf']
>>> 04/22/10 21:17:00 140471479552272 DataStore DataStore.py:1271 DEBUG
>>> UploadFiles: ftps://localhost:
>>> 8006/7f00010133381d7afaba1c148fcabe1007 /home/ercea/Descargas/DSD-
>>> Tutorial-3.pdf [u'/home/ercea/Descargas/ DSD-Tutorial-3.pdf']
>>> 04/22/10 21:17:00 140471479552272 FTPSClient FTPSClient.py:141 DEBUG
>>> Entered FTPSUploadFile: localfile=/home/ercea/Descargas/DSD-
>>> Tutorial-3.pdf url=ftps://localhost:8006/7f00010133381d7afaba1c148fcabe1007
>>> 04/22/10 21:17:00 140471913264880 VenueClientController
>>> VenueClientController.py:862 DEBUG Started thread
>>> 04/22/10 21:17:20 140471517653264 VenueClient VenueClient.py:605
>>> DEBUG Calling Heartbeat, time now: 1271963840
>>> 04/22/10 21:17:20 140471517653264 VenueClient VenueClient.py:629
>>> DEBUG Next Heartbeat needed within 36s
>>> 04/22/10 21:17:20 140471517653264 VenueClient VenueClient.py:639
>>> DEBUG heartBeatCounter = 2
>>> 04/22/10 21:17:56 140471401707792 VenueClient VenueClient.py:605
>>> DEBUG Calling Heartbeat, time now: 1271963876
>>> 04/22/10 21:17:56 140471401707792 VenueClient VenueClient.py:629
>>> DEBUG Next Heartbeat needed within 36s
>>> 04/22/10 21:17:56 140471401707792 VenueClient VenueClient.py:639
>>> DEBUG heartBeatCounter = 3
>>>
>>>
>>> Here increment heartBeatCounter +1.
>>>
>>> Ah, I have this problem, in the same network, on ubuntu system
>>> (Ubuntu 9.10 64 bits), but I've tested on Windows 7 and I can upload
>>> files without problems.
>>>
>>> Regards!
>>>
>>>
>>> El dia 22 abr 2010 18:29, Thomas Uram <turam at mcs.anl.gov> escribió:
>>> Hi Jesus:
>>>
>>> There is probably some indication of the problem in the
>>> VenueClient.log file. If you submit a bug report from the Help menu
>>> after this happens, a recent portion of the VenueClient.log file
>>> will be included with the report, and we can review it. If you want
>>> to look yourself, the log file resides in the following locations
>>> based on platform:
>>> MacOS, Linux: ~/.AccessGrid3/Logs
>>> Windows: c:\documents and settings\username\application data
>>> \AccessGrid3\Logs (adjusted for your locale, of course)
>>> Tom
>>>
>>> On Apr 22, 2010, at 11:11 AM, Jesus Cea Oliva wrote:
>>>
>>> Hi list again, sorry for my constants problems xD.
>>>
>>> I'm trying upload files to our VenueServer and my problem is
>>> strange. Firstly, I've created a VenueServer locally, for testing.
>>>
>>> Well, when I try to upload a file, I can see the message on the
>>> VenueClient "Uploading file X" and a progressbar, but, when the
>>> progressbar is near to the end, stops and list of files (Data
>>> section) don't refresh. So I beleived that the file can't upload.
>>>
>>> But, really, the file has uploaded, because I can see the folder
>>> Data of my local VenueServer, and the file is copied in this folder.
>>> In this case, if I reset VenueServer and VenueClient, then, I can
>>> see the files in Data section.
>>>
>>> So I think that the file uploads fine but, the list of files don't
>>> refresh and the file uploaded don't appears.
>>>
>>> I must say that, sometimes, I can upload a file so I don't know
>>> whats really the problem (ports closed? firewall? path or name
>>> files? file size?) .
>>>
>>> Thanks again and apologize for any inconveniences
>>>
>>> Regards!
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> <VenueClient.log>
>>>
>>> Christoph Willing +61 7 3365 8316
>>> QCIF Access Grid Manager
>>> University of Queensland
>>>
>>> Christoph Willing +61 7 3365 8316
>>> QCIF Access Grid Manager
>>> University of Queensland
>>>
>>>
>>
>> Christoph Willing                       +61 7 3365 8316
>> QCIF Access Grid Manager
>> University of Queensland
>>
>

Christoph Willing                       +61 7 3365 8316
QCIF Access Grid Manager
University of Queensland