[Swift-devel] Re: coaster features

[Zhao Zhang]

Thanks, Mihael. I am stating that bug again.

I wrote a swift script to test the coaster-data-provider.
[zzhang at communicado 1KB]$ cat data-test.swift
type Mol {}
type DOCKOut {}

app (DOCKOut ofile) cat (Mol molfile)
{
   cat @filename(molfile) stdout=@filename(ofile);
}

Mol texts[] <filesys_mapper;location="input/", suffix=".mol2">;

doall(Mol texts[])
{
  foreach p in texts {
    DOCKOut r <regexp_mapper; 
             source=@p, 
             match="input/(.*)\\.mol2", 
             transform=@strcat("result/", "\\1", ".out") 
    >;
   
    (r) = cat(p);
  }
}
 
// Main
doall(texts);


It copies a file "D00001.mol2", then redirects the file to "D00001.out", 
"D00001.out" is taken as the output file.

I have two test cases on CI netowrk,
/home/zzhang/swift_coaster/cog/modules/swift/tests/sites/swift_data_test/1KB 
and
/home/zzhang/swift_coaster/cog/modules/swift/tests/sites/swift_data_test/1MB
If you want to repeat the test, you will also need
/home/zzhang/swift_coaster/cog/modules/swift/tests/sites/tc.data and
/home/zzhang/swift_coaster/cog/modules/swift/tests/sites/coaster_test/AGLT2.xml
[zzhang at communicado 1MB]$ cat ../../coaster_test/AGLT2.xml
<config>
  <!-- AGLT2 -->
  <pool handle="AGLT2" >
    <filesystem provider="coaster" url="gt2://gate01.aglt2.org" />
    <execution  provider="coaster" url="gate01.aglt2.org" 
jobManager="gt2:gt2:condor"/>
    <workdirectory >/atlas/data08/OSG/DATA/osg/tmp/AGLT2</workdirectory>
  </pool>
</config>


So, I ran the test twice with those two cases.
[zzhang at communicado 1KB]$ swift -sites.file ../../coaster_test/AGLT2.xml 
-tc.file ../../tc.data data-test.swift
Swift svn swift-r2896 cog-r2392

RunID: 20090518-1355-dksv64l5
Progress:
Progress:  Stage in:1
Progress:  Submitting:1
Progress:  Submitted:1
Progress:  Submitted:1
Progress:  Submitted:1
Progress:  Active:1
Final status:  Finished successfully:1
Cleaning up...
Shutting down service at https://192.41.230.11:41507
Got channel MetaChannel: 7132795 -> GSSSChannel-null(1)
- Done
[zzhang at communicado input]$ cksum D00001.mol2
2023409842 1054 D00001.mol2
[zzhang at communicado input]$ cksum ../result/D00001.out
2023409842 1054 ../result/D00001.out

The 1KB test case is fine. Then I tried 1MB test case

[zzhang at communicado 1MB]$ swift -sites.file ../../coaster_test/AGLT2.xml 
-tc.file ../../tc.data data-test.swift
Swift svn swift-r2896 cog-r2392
RunID: 20090518-1357-ad024vu1
Progress:
Progress:  Stage in:1
Progress:  Submitting:1
Progress:  Submitted:1
Progress:  Active:1
Progress:  Finished successfully:1
Final status:  Finished successfully:1
Cleaning up...
Shutting down service at https://192.41.230.11:47431
Got channel MetaChannel: 8676252 -> GSSSChannel-null(1)
- Done
[zzhang at communicado 1MB]$ cksum input/D00001.mol2
1748450002 1000000 input/D00001.mol2
[zzhang at communicado 1MB]$ cksum result/D00001.out
4159559983 1000000 result/D00001.out

as you could tell from the check sum, the input and output are not the 
same. I repeated the 1MB test case, the same
thing happened
[zzhang at communicado 1MB]$ swift -sites.file ../../coaster_test/AGLT2.xml 
-tc.file ../../tc.data data-test.swift
Swift svn swift-r2896 cog-r2392

RunID: 20090518-1629-1gef4tm8
Progress:
Progress:  Stage in:1
Progress:  Submitting:1
Progress:  Submitted:1
Progress:  Active:1
Progress:  Finished successfully:1
Final status:  Finished successfully:1
Cleaning up...
Shutting down service at https://192.41.230.11:43400
Got channel MetaChannel: 31518612 -> GSSSChannel-null(1)
- Done
[zzhang at communicado 1MB]$ cksum input/D00001.mol2
1748450002 1000000 input/D00001.mol2
[zzhang at communicado 1MB]$ cksum result/D00001.out
2414944118 1000000 result/D00001.out


zhao


Mihael Hategan wrote:
> On Mon, 2009-05-18 at 14:08 -0500, Zhao Zhang wrote:
>   
>>>>>       
>>>>>           
>>>> This is the "<filesystem provider="coaster" url="gt2://grid.myhost.org" 
>>>> />" from http://www.ci.uchicago.edu/swift/guides/userguide/coasters.php.
>>>>     
>>>>         
>>> Right. The coaster fs provider. Glen found some data corruption with it
>>> when transferring files larger than a few KB. So it's only somewhat
>>> working.
>>>   
>>>       
>> Has this feature been fixed?
>>     
>
> No.
>
>   
>>  I found it again in my data movement test 
>> on OSG site.
>>     
>
> Can you be more specific?
>
> In general "feature X doesn't work properly" is not very useful. I
> recommend reading this:
>
> http://www.chiark.greenend.org.uk/~sgtatham/bugs.html
>
>
>