[Swift-devel] Re: coaster features

[Mihael Hategan]

Excellent! Please put this in bugzilla, and refer to the bug number
whenever necessary.

Mihael

On Mon, 2009-05-18 at 16:32 -0500, Zhao Zhang wrote:
> Thanks, Mihael. I am stating that bug again.
> 
> I wrote a swift script to test the coaster-data-provider.
> [zzhang at communicado 1KB]$ cat data-test.swift
> type Mol {}
> type DOCKOut {}
> 
> app (DOCKOut ofile) cat (Mol molfile)
> {
>    cat @filename(molfile) stdout=@filename(ofile);
> }
> 
> Mol texts[] <filesys_mapper;location="input/", suffix=".mol2">;
> 
> doall(Mol texts[])
> {
>   foreach p in texts {
>     DOCKOut r <regexp_mapper; 
>              source=@p, 
>              match="input/(.*)\\.mol2", 
>              transform=@strcat("result/", "\\1", ".out") 
>     >;
>    
>     (r) = cat(p);
>   }
> }
>  
> // Main
> doall(texts);
> 
> 
> It copies a file "D00001.mol2", then redirects the file to "D00001.out", 
> "D00001.out" is taken as the output file.
> 
> I have two test cases on CI netowrk,
> /home/zzhang/swift_coaster/cog/modules/swift/tests/sites/swift_data_test/1KB 
> and
> /home/zzhang/swift_coaster/cog/modules/swift/tests/sites/swift_data_test/1MB
> If you want to repeat the test, you will also need
> /home/zzhang/swift_coaster/cog/modules/swift/tests/sites/tc.data and
> /home/zzhang/swift_coaster/cog/modules/swift/tests/sites/coaster_test/AGLT2.xml
> [zzhang at communicado 1MB]$ cat ../../coaster_test/AGLT2.xml
> <config>
>   <!-- AGLT2 -->
>   <pool handle="AGLT2" >
>     <filesystem provider="coaster" url="gt2://gate01.aglt2.org" />
>     <execution  provider="coaster" url="gate01.aglt2.org" 
> jobManager="gt2:gt2:condor"/>
>     <workdirectory >/atlas/data08/OSG/DATA/osg/tmp/AGLT2</workdirectory>
>   </pool>
> </config>
> 
> 
> So, I ran the test twice with those two cases.
> [zzhang at communicado 1KB]$ swift -sites.file ../../coaster_test/AGLT2.xml 
> -tc.file ../../tc.data data-test.swift
> Swift svn swift-r2896 cog-r2392
> 
> RunID: 20090518-1355-dksv64l5
> Progress:
> Progress:  Stage in:1
> Progress:  Submitting:1
> Progress:  Submitted:1
> Progress:  Submitted:1
> Progress:  Submitted:1
> Progress:  Active:1
> Final status:  Finished successfully:1
> Cleaning up...
> Shutting down service at https://192.41.230.11:41507
> Got channel MetaChannel: 7132795 -> GSSSChannel-null(1)
> - Done
> [zzhang at communicado input]$ cksum D00001.mol2
> 2023409842 1054 D00001.mol2
> [zzhang at communicado input]$ cksum ../result/D00001.out
> 2023409842 1054 ../result/D00001.out
> 
> The 1KB test case is fine. Then I tried 1MB test case
> 
> [zzhang at communicado 1MB]$ swift -sites.file ../../coaster_test/AGLT2.xml 
> -tc.file ../../tc.data data-test.swift
> Swift svn swift-r2896 cog-r2392
> RunID: 20090518-1357-ad024vu1
> Progress:
> Progress:  Stage in:1
> Progress:  Submitting:1
> Progress:  Submitted:1
> Progress:  Active:1
> Progress:  Finished successfully:1
> Final status:  Finished successfully:1
> Cleaning up...
> Shutting down service at https://192.41.230.11:47431
> Got channel MetaChannel: 8676252 -> GSSSChannel-null(1)
> - Done
> [zzhang at communicado 1MB]$ cksum input/D00001.mol2
> 1748450002 1000000 input/D00001.mol2
> [zzhang at communicado 1MB]$ cksum result/D00001.out
> 4159559983 1000000 result/D00001.out
> 
> as you could tell from the check sum, the input and output are not the 
> same. I repeated the 1MB test case, the same
> thing happened
> [zzhang at communicado 1MB]$ swift -sites.file ../../coaster_test/AGLT2.xml 
> -tc.file ../../tc.data data-test.swift
> Swift svn swift-r2896 cog-r2392
> 
> RunID: 20090518-1629-1gef4tm8
> Progress:
> Progress:  Stage in:1
> Progress:  Submitting:1
> Progress:  Submitted:1
> Progress:  Active:1
> Progress:  Finished successfully:1
> Final status:  Finished successfully:1
> Cleaning up...
> Shutting down service at https://192.41.230.11:43400
> Got channel MetaChannel: 31518612 -> GSSSChannel-null(1)
> - Done
> [zzhang at communicado 1MB]$ cksum input/D00001.mol2
> 1748450002 1000000 input/D00001.mol2
> [zzhang at communicado 1MB]$ cksum result/D00001.out
> 2414944118 1000000 result/D00001.out
> 
> 
> zhao
> 
> 
> Mihael Hategan wrote:
> > On Mon, 2009-05-18 at 14:08 -0500, Zhao Zhang wrote:
> >   
> >>>>>       
> >>>>>           
> >>>> This is the "<filesystem provider="coaster" url="gt2://grid.myhost.org" 
> >>>> />" from http://www.ci.uchicago.edu/swift/guides/userguide/coasters.php.
> >>>>     
> >>>>         
> >>> Right. The coaster fs provider. Glen found some data corruption with it
> >>> when transferring files larger than a few KB. So it's only somewhat
> >>> working.
> >>>   
> >>>       
> >> Has this feature been fixed?
> >>     
> >
> > No.
> >
> >   
> >>  I found it again in my data movement test 
> >> on OSG site.
> >>     
> >
> > Can you be more specific?
> >
> > In general "feature X doesn't work properly" is not very useful. I
> > recommend reading this:
> >
> > http://www.chiark.greenend.org.uk/~sgtatham/bugs.html
> >
> >
> >