[Swift-devel] Work on generalizing OSG scripts for a future Swift release

[Michael Wilde]

Ketan, this message is to follow up on the longer-term task of making Swift's OSG execution capability end-user ready. Im cc'ing swift-devel so that I and others on the team can help you on various aspects of this. 


You can find the versions that I had started to work on, on the CI network here: 


/home/wilde/swift/lab/osg/allantools/{ pool_coaster,site_gen} 

site_gen has tools to generate the sites.xml file based on OSG configuration services (ReSS on Condor). 


pool_coaster has the script that starts coaster workers on all the sites, using what we call the "Queue-N" algorithm. 


There is a README file in pool_coaster 


Basically the way it works is you cd to site_gen and run gen_goodsites.sh 


Then you cd to pool_coaster and run start_services.sh 


But I think there are several other manual steps involved as decribed in the README 


Before you start this work, you should test a few other Swift mechanisms for running simpler manual coaster pools. You can do that while you wait for your OSG cert to be approved and then for your OSG Engage VO registration to be approved and to propagate to all the OSG sites. That will take about a week, during which its good to learn (and help us document) what end-users need to know about how the coaster mechanism works in manually-run pools (and, in its automated mode as well). 


A paper that Mihael has in progress on coasters is in svn at: 
URL: https://svn.ci.uchicago.edu/svn/swift/2010CloudCom-coasters 


We will need to help you understand the scripts above. And the core scripts in both directories are written in Ruby (and some in Karajan) so there is a lot to learn! But you can start looking at them and send me any questions about the (hopefully Allan and others can get involved as well depending on their time availability). 


We need to find from Justin, Sarah, and Tim (who has done similar things in SwiftR) what the latest scripts we have for starting coasters in various configurations are, and which are heading towards end-user readiness. 


- Mike 


----- Original Message -----


Congratulations Allan. 


Mike discussed your strategy for obtaining upto 2K OSG processors the other day which, I thought is awesome. 


Would you be presenting your work somewhere? It would be nice to attend. 


Cheers, 
Ketan 


On Fri, Mar 4, 2011 at 10:35 PM, Michael Wilde < wilde at mcs.anl.gov > wrote: 


Hi All, 

Allan's just completed MS thesis (advised by Ian and Dan) is on a workflow that includes hundreds of thousands of small tasks among many large ones, and is a good candidate to study and perform in ExM. 

Its on the ExM web Documents page and at: 

https://sites.google.com/site/exmproject/documents/AEspinosa.MSThesis.2011.0304_main.pdf 

I'm sure Allan and Dan would be happy to discuss it with us. 

Congrats, Allan, on completing your thesis! 

- Mike 

_______________________________________________ 
ExM mailing list 
ExM at lists.mcs.anl.gov 
https://lists.mcs.anl.gov/mailman/listinfo/exm 




-- 
Michael Wilde 
Computation Institute, University of Chicago 
Mathematics and Computer Science Division 
Argonne National Laboratory 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110305/3efffc59/attachment.html>