From iraicu at cs.uchicago.edu Wed Jan 14 11:30:32 2009 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Wed, 14 Jan 2009 11:30:32 -0600 Subject: [Swift-user] CFP: IEEE 2009 Third International Workshop on Scientific Workflows (SWF 2009) Message-ID: <496E2138.7060104@cs.uchicago.edu> IEEE 2009 Third International Workshop on Scientific Workflows (SWF 2009) Los Angeles, USA, July 7, 2009 http://www.servicescongress.org/2009/1/swf-2009.html In conjunction with IEEE International Conference on Web Services (ICWS 2009) Call for Papers Today, many scientific discoveries are achieved through complex and distributed scientific computations that are represented and structured as scientific workflows. User friendly scientific workflow systems are increasingly being developed to enable e-scientists to integrate, structure, and orchestrate various local or remote data and service resources to perform various in silico experiments to produce interesting scientific discovery. The critical role of scientific workflows in cyberinfrastructure bas been recognized by a recent NSF workshop on the challenges of scientific workflows in May 2006, which concluded that ?workflows should become first-class entities in cyberinfrastructure architecture. For domain scientists, they are important because workflows document and manage the increasingly complex processes involved in exploration and discovery through computations. For computer scientists, workflows provide a formal and declarative representation of complex distributed computations that must be managed efficiently through their lifecycle from assembly, to execution, to sharing.? Authors are invited to submit regular papers (8 pages), short papers (4 pages), and demo papers (2 pages) that show original unpublished research results in all areas of scientific workflows. Topics of interest are listed below; however, submissions on all aspects of scientific workflows are welcome. For demo papers, at least one author is expected to present a demo in the workshop during the demo session, special arrangement will be made to meet the need of the authors. Accepted SWF 2009 papers will be included in the proceedings of SERVICES 2009 (Part I), which will be published by IEEE Computer Society Press. Topics ? Architecture, model, and language ? Provenance management ? Task management ? Workflow scheduling ? Data product management ? Monitoring and failure handling ? Service, Grid, and Cloud workflows ? Scientific workflow composition ? Scientific workflow security ? Modeling, simulation, analysis ? Scalability, reliability, extensibility ? Scientific workflow applications Important dates February 16, 2009, paper submission; March 20, 2009, notification; April 10, 2009, camera-ready version due. Workshop organizers Workshop chairs: Shiyong Lu, Wayne State University, shiyong at wayne.edu; Calton Pu, Georgia Tech Publicity chairs: Yong Zhao, Microsoft Corporation; Ilkay Altintas, San Diego Supercomputer Center Publication chair: Cui Lin, Wayne State University Previous SWF workshops http://www.cs.wayne.edu/~shiyong/swf -- =================================================== Ioan Raicu Ph.D. Candidate =================================================== Distributed Systems Laboratory Computer Science Department University of Chicago 1100 E. 58th Street, Ryerson Hall Chicago, IL 60637 =================================================== Email: iraicu at cs.uchicago.edu Web: http://www.cs.uchicago.edu/~iraicu http://dev.globus.org/wiki/Incubator/Falkon http://dsl-wiki.cs.uchicago.edu/index.php/Main_Page =================================================== =================================================== -------------- next part -------------- A non-text attachment was scrubbed... Name: SWF09_CFP.pdf Type: application/pdf Size: 33125 bytes Desc: not available URL: From iraicu at cs.uchicago.edu Thu Jan 15 12:19:28 2009 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Thu, 15 Jan 2009 12:19:28 -0600 Subject: [Swift-user] CFP: Workshop on Large-Scale System and Application Performance (LSAP2009) Message-ID: <496F7E30.1020305@cs.uchicago.edu> Call for Papers --------------- Workshop on Large-Scale System and Application Performance (LSAP2009) In conjunction with the 18-th International Symposium on High Performance Distributed Computing (HPDC-18) Munich, Germany, June 9 or 10, 2009 http://www.lsap.org *** Submission deadline: March 1, 2009 *** MISSION Over the last decade, computer systems and applications in everyday use have grown to unprecedented scales. Large clusters serving millions of search requests per day, grids executing large workflows and parameter sweeps consisting of thousands of jobs, and supercomputers running complex e-science applications, have now hundreds of thousands of processing cores. In addition, clouds are quickly emerging as a large-scale computing infrastructure. Peer-to-peer systems and centralized video distribution systems that dominate the internet and complicated internet applications such as massive multiplayer online games are used by millions of people every day. In view of this tremendous growth, understanding the performance of large-scale computer systems and applications has become vital to institutional, commercial, and private interests. This workshop solicits original papers on performance evaluation methods, tools, and studies focusing on the challenges of large scale, such as decentralization, predictable performance, reliability, and scalability. It aims to bring together system designers and researchers involved with the modeling and performance evaluation of large-scale systems and applications. Topics of interest include, but are not limited to: - Performance aspects of large-scale systems - Performance aspects of large-scale applications - Performance-oriented properties such as availability, reliability, and scalability - Workload characterization and modeling - Mathematical modeling and analysis methods - Simulation methods and tools - Measurement methods and tools - Performance case studies SUBMISSION GUIDELINES Submitted papers should be limited to 8 pages (including tables, images, and references) and formatted according to the ACM SIGS Style. Use the official HPDC conference submission site to submit your paper; only pdf format is accepted. All papers will receive at least three reviews. Submission implies the willingness of at least one of the authors to register for the workshop and present the paper. The authors of the best paper in the workshop will receive a best-paper award. PROCEEDINGS The proceedings of the workshop will be published by ACM. IMPORTANT DATES Submission deadline: March 1, 2009 Author notification: March 31, 2009 Final papers due: TBA (in April, 2009) Workshop: June 9 or 10, 2009 SUBMISSION SITE Official HPDC conference submission site, https://ssl.linklings.net/conferences/hpdc/ WORKSHOP WEBSITE www.lsap2009.org PROGRAM CO-CHAIRS Dick Epema, Delft University of Technology, NL, d.h.j.epema at tudelft.nl Jose Moreira, IBM T.J. Watson Research Lab, USA, jmoreira at us.ibm.com Carey Williamson, University of Calgary, Canada, carey at cpsc.ucalgary.ca PROGRAM COMMITTEE Martin Arlitt, HP Labs, USA, and University of Calgary, CA Peter Buchholz, University of Dortmund, Germany Jon Howell, Microsoft Research, USA Adriana Iamnitchi, University of South Florida, USA Alexandru Iosup, Delft University of Technology, NL Evgenia Smirni, College of William and Mary, USA Swami Sivasubramanian, Amazon, USA Allen Snavely, University of California, San Diego, USA Denis Trystram, Labaratoire d'Informatique de Grenoble, FR CONTACT For further information please contact Dick Epema at d.h.j.epema at tudelft.nl. -- =================================================== Ioan Raicu Ph.D. Candidate =================================================== Distributed Systems Laboratory Computer Science Department University of Chicago 1100 E. 58th Street, Ryerson Hall Chicago, IL 60637 =================================================== Email:iraicu at cs.uchicago.edu Web:http://www.cs.uchicago.edu/~iraicu http://dev.globus.org/wiki/Incubator/Falkon http://dsl-wiki.cs.uchicago.edu/index.php/Main_Page =================================================== =================================================== -------------- next part -------------- A non-text attachment was scrubbed... Name: lsap2009_cfp.pdf Type: application/pdf Size: 45831 bytes Desc: not available URL: From eduardo.ogasawara at yahoo.com.br Fri Jan 16 12:34:21 2009 From: eduardo.ogasawara at yahoo.com.br (Eduardo S. Ogasawara) Date: Fri, 16 Jan 2009 16:34:21 -0200 Subject: [Swift-user] Swift license model Message-ID: <985EF559C42445978BA089A6F81F5C24@mp424> Dear users, I would like to know what is the license model used for Swift. I know that it is open source, but I did not find any information about the exactly type of license. Sincerely, Eduardo Ogasawara http://www.cos.ufrj.br/~ogasawara -------------- next part -------------- An HTML attachment was scrubbed... URL: From benc at hawaga.org.uk Fri Jan 16 12:55:45 2009 From: benc at hawaga.org.uk (Ben Clifford) Date: Fri, 16 Jan 2009 18:55:45 +0000 (GMT) Subject: [Swift-user] Swift license model In-Reply-To: <985EF559C42445978BA089A6F81F5C24@mp424> References: <985EF559C42445978BA089A6F81F5C24@mp424> Message-ID: On Fri, 16 Jan 2009, Eduardo S. Ogasawara wrote: > I would like to know what is the license model used for Swift. > > I know that it is open source, but I did not find any information about > the exactly type of license. It is licenced under something that is very much the Apache v2 licence. For more information, see this page: http://dev.globus.org/wiki/Licensing_and_Contribution_Guidance -- From wwj at ci.uchicago.edu Wed Jan 21 13:50:05 2009 From: wwj at ci.uchicago.edu (Wenjun Wu) Date: Wed, 21 Jan 2009 13:50:05 -0600 Subject: [Swift-user] exceptions in running many data files Message-ID: <49777C6D.1090307@ci.uchicago.edu> Hello, I often get the following exception while running thousands of files: Progress: java.io.IOException: Bad file descriptor at java.io.FileInputStream.available(Native Method) at java.io.BufferedInputStream.available(BufferedInputStream.java:337) at org.griphyn.vdl.karajan.InHook.run(InHook.java:37) at java.lang.Thread.run(Thread.java:534) My workfow script looks like this: seqfile trunks[] ; foreach file in trunks { seqfile outvid ; outvid = runAnalysis(file); } Could anyone give any suggestion about how to fix such an exception? Thanks, Wenjun From benc at hawaga.org.uk Wed Jan 21 14:14:29 2009 From: benc at hawaga.org.uk (Ben Clifford) Date: Wed, 21 Jan 2009 20:14:29 +0000 (GMT) Subject: [Swift-user] exceptions in running many data files In-Reply-To: <49777C6D.1090307@ci.uchicago.edu> References: <49777C6D.1090307@ci.uchicago.edu> Message-ID: On Wed, 21 Jan 2009, Wenjun Wu wrote: > I often get the following exception while running thousands of files: There were bug-fixes made in this area in r2343 (2008-11-18), r2352 (2008-12-02) and r2356 (2008-12-04). Based on your stack trace, you aren't running the latest code; you might try that (or if you wait until Friday you can try swift 0.8 release candidate 1) -- From wilde at mcs.anl.gov Mon Jan 26 12:37:15 2009 From: wilde at mcs.anl.gov (Michael Wilde) Date: Mon, 26 Jan 2009 12:37:15 -0600 Subject: [Swift-user] Expanding arrays in app function command lines Message-ID: <497E02DB.8040904@mcs.anl.gov> I want to pass an array of parameter strings (name/value pairs) to an app and have the parameters get expanded on the app command line in the order that they are indexed in the array. What seems to happen instead is that they are being passed in semi-random order (but seems to be the same every time). The app cmd line template is: runoops @fasta @secseq @pdb @pdt @rmsd jobnum cfgParams[*] stdout=@log; The cfgParams formal parameter declared as: string cfgParams[] is passed the array "config" as a value: string config []; config[0] = "TEMP UPDATE INTERVAL"; config[1]="10"; config[2] = "SMOOTH DEVIATION COEFFICIENT"; config[3]="0.80001"; What gets sent to the command line is: [0.80001, SMOOTH DEVIATION COEFFICIENT, TEMP UPDATE INTERVAL, 10] This seems to come out in this order (3,2,0,1) every time, but what I was expecting was: [TEMP UPDATE INTERVAL, 10, SMOOTH DEVIATION COEFFICIENT, 0.80001] Actually I wanted each param to be passed as a separate command-line arg, but the current format is ok. In this particular case I can work around the random order by putting the name and value in the same string and break it up in the application script. But in general, the ability to expand an array of strings into the command line is useful and I think should be done in array member order. Can/should this be changed to behave like that? The current version of the full script is attached. - Mike -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: oops3.swift URL: From benc at hawaga.org.uk Mon Jan 26 12:41:21 2009 From: benc at hawaga.org.uk (Ben Clifford) Date: Mon, 26 Jan 2009 18:41:21 +0000 (GMT) Subject: [Swift-user] Expanding arrays in app function command lines In-Reply-To: <497E02DB.8040904@mcs.anl.gov> References: <497E02DB.8040904@mcs.anl.gov> Message-ID: On Mon, 26 Jan 2009, Michael Wilde wrote: > But in general, the ability to expand an array of strings into the command > line is useful and I think should be done in array member order. > Can/should this be changed to behave like that? yes, this is something I've been meaning to work on. It would tidy up some stuff in the type system too, I think. -- From wilde at mcs.anl.gov Mon Jan 26 12:48:51 2009 From: wilde at mcs.anl.gov (Michael Wilde) Date: Mon, 26 Jan 2009 12:48:51 -0600 Subject: [Swift-user] Expanding arrays in app function command lines In-Reply-To: References: <497E02DB.8040904@mcs.anl.gov> Message-ID: <497E0593.6030401@mcs.anl.gov> Very good. I have not yet tried it, but would also be interesting to discuss how a struct should be expanded. That might make passing groups of file names nice and compact. I dont know if this does something useful at the moment, but will try. On 1/26/09 12:41 PM, Ben Clifford wrote: > On Mon, 26 Jan 2009, Michael Wilde wrote: > >> But in general, the ability to expand an array of strings into the command >> line is useful and I think should be done in array member order. > >> Can/should this be changed to behave like that? > > yes, this is something I've been meaning to work on. > > It would tidy up some stuff in the type system too, I think. > From wwj at ci.uchicago.edu Tue Jan 27 00:40:51 2009 From: wwj at ci.uchicago.edu (Wenjun Wu) Date: Tue, 27 Jan 2009 00:40:51 -0600 Subject: [Swift-user] exceptions in running many data files In-Reply-To: References: <49777C6D.1090307@ci.uchicago.edu> Message-ID: <497EAC73.5020108@ci.uchicago.edu> After I updated to the new version, the problem was gone. Thanks, Wenjun > On Wed, 21 Jan 2009, Wenjun Wu wrote: > > >> I often get the following exception while running thousands of files: >> > > There were bug-fixes made in this area in r2343 (2008-11-18), r2352 > (2008-12-02) and r2356 (2008-12-04). Based on your stack trace, you aren't > running the latest code; you might try that (or if you wait until Friday > you can try swift 0.8 release candidate 1) > > From benc at hawaga.org.uk Tue Jan 27 04:52:16 2009 From: benc at hawaga.org.uk (Ben Clifford) Date: Tue, 27 Jan 2009 10:52:16 +0000 (GMT) Subject: [Swift-user] Expanding arrays in app function command lines In-Reply-To: <497E02DB.8040904@mcs.anl.gov> References: <497E02DB.8040904@mcs.anl.gov> Message-ID: On Mon, 26 Jan 2009, Michael Wilde wrote: > The app cmd line template is: > > runoops @fasta @secseq @pdb @pdt @rmsd jobnum cfgParams[*] stdout=@log; > config[0] = "TEMP UPDATE INTERVAL"; config[1]="10"; > config[2] = "SMOOTH DEVIATION COEFFICIENT"; config[3]="0.80001"; > > What gets sent to the command line is: > > [0.80001, SMOOTH DEVIATION COEFFICIENT, TEMP UPDATE INTERVAL, 10] Looking at this, and aside from implementing better functionality, I think there has been a change of behaviour here from earlier versions of Swift. It used to be, I think, that config[*] would have generated a sequence of commandline parameters when used on the command line (so pretty much what you wanted). There's no language-behaviour/ test for [*] though. -- From benc at hawaga.org.uk Tue Jan 27 05:54:44 2009 From: benc at hawaga.org.uk (Ben Clifford) Date: Tue, 27 Jan 2009 11:54:44 +0000 (GMT) Subject: [Swift-user] Expanding arrays in app function command lines In-Reply-To: References: <497E02DB.8040904@mcs.anl.gov> Message-ID: On Tue, 27 Jan 2009, Ben Clifford wrote: > Looking at this, and aside from implementing better functionality, I think > there has been a change of behaviour here from earlier versions of Swift. > It used to be, I think, that config[*] would have generated a sequence of > commandline parameters when used on the command line (so pretty much what > you wanted). There's no language-behaviour/ test for [*] though. Well, I just tried with 0.3rc1 from october 2007, and it has the same behaviour. so maybe I am misremembering... -- From benc at hawaga.org.uk Tue Jan 27 10:11:01 2009 From: benc at hawaga.org.uk (Ben Clifford) Date: Tue, 27 Jan 2009 16:11:01 +0000 (GMT) Subject: [Swift-user] Expanding arrays in app function command lines In-Reply-To: References: <497E02DB.8040904@mcs.anl.gov> Message-ID: On Mon, 26 Jan 2009, Ben Clifford wrote: > > But in general, the ability to expand an array of strings into the command > > line is useful and I think should be done in array member order. > > > Can/should this be changed to behave like that? > > yes, this is something I've been meaning to work on. http://www.ci.uchicago.edu/~benc/tmp/argstar.patch will allow you to specify arrays of strings directly as parameters. http://www.ci.uchicago.edu/~benc/tmp/strarray.swift demonstrates the use of such. You can apply that, or you can wait a few days more me to test it more thoroughly and commit. Note that instead of saying x[*] you would say x without further modifiers. -- From wilde at mcs.anl.gov Tue Jan 27 10:26:54 2009 From: wilde at mcs.anl.gov (Michael Wilde) Date: Tue, 27 Jan 2009 10:26:54 -0600 Subject: [Swift-user] Expanding arrays in app function command lines In-Reply-To: References: <497E02DB.8040904@mcs.anl.gov> Message-ID: <497F35CE.8070709@mcs.anl.gov> Thanks, Ben. The example looks good - that seems the natural way to express it. Might want to specify/test the following: - array of other scalar types, coerced into string in the cmd line - struct? (expand the fields in source-code order)? - array of struct etc Or for now just fail the cases that are not easy to support? The struct case would be handy for the oops code that is using the parameter-string array case: the app returns 5 files; would be nice to specify them as a struct type and pass a single argument for results called "oops_results". Same for its 3 input files. - Mike On 1/27/09 10:11 AM, Ben Clifford wrote: > On Mon, 26 Jan 2009, Ben Clifford wrote: > >>> But in general, the ability to expand an array of strings into the command >>> line is useful and I think should be done in array member order. >>> Can/should this be changed to behave like that? >> yes, this is something I've been meaning to work on. > > http://www.ci.uchicago.edu/~benc/tmp/argstar.patch > > will allow you to specify arrays of strings directly as parameters. > > http://www.ci.uchicago.edu/~benc/tmp/strarray.swift demonstrates the use > of such. > > You can apply that, or you can wait a few days more me to test it more > thoroughly and commit. > > Note that instead of saying x[*] you would say x without further > modifiers. > From benc at hawaga.org.uk Tue Jan 27 10:29:27 2009 From: benc at hawaga.org.uk (Ben Clifford) Date: Tue, 27 Jan 2009 16:29:27 +0000 (GMT) Subject: [Swift-user] Expanding arrays in app function command lines In-Reply-To: <497F35CE.8070709@mcs.anl.gov> References: <497E02DB.8040904@mcs.anl.gov> <497F35CE.8070709@mcs.anl.gov> Message-ID: On Tue, 27 Jan 2009, Michael Wilde wrote: > - array of other scalar types, coerced into string in the cmd line I think that might already happen. Everything apart from struct has a pretty natural expansion into string args. The source-code order of fields for structs is probably the best, unless someone comes up with a better idea. -- From hategan at mcs.anl.gov Tue Jan 27 21:29:25 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 27 Jan 2009 21:29:25 -0600 Subject: [Swift-user] Expanding arrays in app function command lines In-Reply-To: References: <497E02DB.8040904@mcs.anl.gov> <497F35CE.8070709@mcs.anl.gov> Message-ID: <1233113365.2159.23.camel@localhost> On Tue, 2009-01-27 at 16:29 +0000, Ben Clifford wrote: > On Tue, 27 Jan 2009, Michael Wilde wrote: > > > - array of other scalar types, coerced into string in the cmd line > > I think that might already happen. > > Everything apart from struct has a pretty natural expansion into string > args. The source-code order of fields for structs is probably the best, > unless someone comes up with a better idea. Though I think it may be one of those "shoot yourself in the feet" features. In a sense I do like the a[*] syntax because it makes it very clear that there is a variable number of things being passed. From benc at hawaga.org.uk Wed Jan 28 01:55:39 2009 From: benc at hawaga.org.uk (Ben Clifford) Date: Wed, 28 Jan 2009 07:55:39 +0000 (GMT) Subject: [Swift-user] Expanding arrays in app function command lines In-Reply-To: <1233113365.2159.23.camel@localhost> References: <497E02DB.8040904@mcs.anl.gov> <497F35CE.8070709@mcs.anl.gov> <1233113365.2159.23.camel@localhost> Message-ID: On Tue, 27 Jan 2009, Mihael Hategan wrote: > In a sense I do like the a[*] syntax because it makes it very clear that > there is a variable number of things being passed. Should you be compelled to put it after @filenames? (eg @filenames(whatever)[*]) to indicate that you are passing multiple things? It also looks icky from a type system perspective - what is the type of a[*] and how is that distinct from a? Its a (undocumented and unchecked) mysterious ordered sequence type that you can only use specifically in app parameters and a few other places, that there is not much to handle and that you should pretend doesn't exist... -- From hategan at mcs.anl.gov Wed Jan 28 09:45:32 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 28 Jan 2009 09:45:32 -0600 Subject: [Swift-user] Expanding arrays in app function command lines In-Reply-To: References: <497E02DB.8040904@mcs.anl.gov> <497F35CE.8070709@mcs.anl.gov> <1233113365.2159.23.camel@localhost> Message-ID: <1233157532.5004.3.camel@localhost> On Wed, 2009-01-28 at 07:55 +0000, Ben Clifford wrote: > On Tue, 27 Jan 2009, Mihael Hategan wrote: > > > In a sense I do like the a[*] syntax because it makes it very clear that > > there is a variable number of things being passed. > > Should you be compelled to put it after @filenames? (eg > @filenames(whatever)[*]) to indicate that you are passing multiple things? The "s" in "filenames" seems to be sufficient. Though it probably does the wrong thing in this case. > > It also looks icky from a type system perspective - what is the type of > a[*] and how is that distinct from a? tuple versus array. It's standard issue in ML, Python, and other languages. > Its a (undocumented and unchecked) > mysterious ordered sequence type that you can only use specifically in app > parameters and a few other places, that there is not much to handle and > that you should pretend doesn't exist... > I wouldn't change things. Just mentioned what I thought on the subject. From benc at hawaga.org.uk Wed Jan 28 09:59:19 2009 From: benc at hawaga.org.uk (=?ISO-8859-1?Q?Ben_Clifford?=) Date: Wed, 28 Jan 2009 07:59:19 -0800 (PST) Subject: =?ISO-8859-1?Q?RE:_Re:_[Swift-user]_Expanding_arrays_in_app_funct?= =?ISO-8859-1?Q?ion_command_lines?= Message-ID: <498080d7.0508d00a.694b.785d@mx.google.com> It's not really a tuple in say the Haskell sense- in those, the number of elements is statically defined and are more like anonymous structures where the elements are named by position. -- From hategan at mcs.anl.gov Wed Jan 28 10:40:19 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 28 Jan 2009 10:40:19 -0600 Subject: [Swift-user] Expanding arrays in app function command lines In-Reply-To: <498080d7.0508d00a.694b.785d@mx.google.com> References: <498080d7.0508d00a.694b.785d@mx.google.com> Message-ID: <1233160819.6101.10.camel@localhost> On Wed, 2009-01-28 at 07:59 -0800, Ben Clifford wrote: > It's not really a tuple in say the Haskell sense- in those, the number of elements is statically defined and are more like anonymous structures where the elements are named by position. That's not the issue. I wasn't looking at it from a type validation perspective. We're already doing implicit type coercion there. I wanted some way to make it clear that '"-a" array' will not likely result in what the user expects (i.e. one argument after "-a"). From wilde at mcs.anl.gov Thu Jan 29 11:47:40 2009 From: wilde at mcs.anl.gov (Michael Wilde) Date: Thu, 29 Jan 2009 11:47:40 -0600 Subject: [Swift-user] When to use Coaster filesystem data provider? Message-ID: <4981EBBC.8080700@mcs.anl.gov> Please explain (on this list and/or in the userguide) more about this data provider method: - what are its performance characteristics? - when would one select it vs gridftp? - how does it work? Was this implemented recently, or has it been in the coaster implementation for quite a while? I never heard about it till recent discussions on coaster config issues in sites.xml. From benc at hawaga.org.uk Thu Jan 29 11:51:43 2009 From: benc at hawaga.org.uk (Ben Clifford) Date: Thu, 29 Jan 2009 17:51:43 +0000 (GMT) Subject: [Swift-user] When to use Coaster filesystem data provider? In-Reply-To: <4981EBBC.8080700@mcs.anl.gov> References: <4981EBBC.8080700@mcs.anl.gov> Message-ID: On Thu, 29 Jan 2009, Michael Wilde wrote: > Please explain (on this list and/or in the userguide) more about this > data provider method: It was/is an attempt to get better performance for small files (on the order of bytes/kilobytes). I haven't done any measurements on how it performs vs gridftp, though Mihael may have done. Its been around for a few months. -- From hategan at mcs.anl.gov Thu Jan 29 12:17:18 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 29 Jan 2009 12:17:18 -0600 Subject: [Swift-user] When to use Coaster filesystem data provider? In-Reply-To: <4981EBBC.8080700@mcs.anl.gov> References: <4981EBBC.8080700@mcs.anl.gov> Message-ID: <1233253038.16245.14.camel@localhost> On Thu, 2009-01-29 at 11:47 -0600, Michael Wilde wrote: > Please explain (on this list and/or in the userguide) more about this > data provider method: > > - what are its performance characteristics? Similar to coaster execution in that authentication cost is amortized and high parallelism (say, >128) can be achieved. This means that for small files you can get near 100% bandwidth utilization. > > - when would one select it vs gridftp? When there are many small files involved and you are willing to deal with not-so-well-tested software. > > - how does it work? File data is sent as messages through the same communication library that coasters use. In principle there is one TCP/SSL connection used for two-way tagged messages, so RTT is amortized with high parallelism. > > Was this implemented recently, or has it been in the coaster > implementation for quite a while? It's been there since I was trying to speed up skenny's 64k jobs workflow on ranger. ?See http://mail.ci.uchicago.edu/mailman/private/swift-devel/2008-September/003879.html From benc at hawaga.org.uk Fri Jan 30 08:12:43 2009 From: benc at hawaga.org.uk (Ben Clifford) Date: Fri, 30 Jan 2009 14:12:43 +0000 (GMT) Subject: [Swift-user] Swift 0.8 released. Message-ID: Swift 0.8 is released. Downlaod it at http://www.ci.uchicago.edu/swift/downloads/ In addition to a number of bugfixes and improvements in output and error quality, a new swift-plot-log command is included to make pretty graphical plots of swift runs. The release notes, with more information, are available at: http://www.ci.uchicago.edu/swift/downloads/release-notes-0.8.txt --