From davidk at ci.uchicago.edu Mon Dec 2 14:24:03 2013 From: davidk at ci.uchicago.edu (David Kelly) Date: Mon, 2 Dec 2013 14:24:03 -0600 Subject: [Swift-devel] 0.95 RC1 Message-ID: Hello, I've created an 0.95 branch and a first release candidate. The paths are: svn co https://svn.code.sf.net/p/cogkit/svn/branches/4.1.11/src/cog svn co https://svn.ci.uchicago.edu/svn/vdl2/branches/release-0.95 swift -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidk at ci.uchicago.edu Thu Dec 5 12:56:18 2013 From: davidk at ci.uchicago.edu (David Kelly) Date: Thu, 5 Dec 2013 12:56:18 -0600 Subject: [Swift-devel] 0.95 RC1 In-Reply-To: References: Message-ID: Below is the to-do list for a 0.95 release: - Bug #1139 - issue with not being able to run on OSG in trunk - Renaming of coaster configuration properties ( http://goo.gl/wIbcam ) - A foreachworker script that executes commands on all coaster workers ( http://goo.gl/5YuPP0 ) - Growing and shrinking of pools with persistent coasters - Merge site guide and userguide - Document new configuration settings in userguide - Better error handling of provider failures - Automated testing of 0.95 - Release notes - Deploy modules - Write a test for Mihael's "install things from a tar file" mechanism - Yadu's batch mapping - Make the debug info in the info log more useful (make the deafult log level quieter, less noise) On Mon, Dec 2, 2013 at 2:24 PM, David Kelly wrote: > Hello, > > I've created an 0.95 branch and a first release candidate. The paths are: > > svn co https://svn.code.sf.net/p/cogkit/svn/branches/4.1.11/src/cog > svn co https://svn.ci.uchicago.edu/svn/vdl2/branches/release-0.95 swift > -------------- next part -------------- An HTML attachment was scrubbed... URL: From iraicu at cs.iit.edu Sat Dec 7 13:40:18 2013 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Sat, 07 Dec 2013 13:40:18 -0600 Subject: [Swift-devel] CFP: The 23nd International ACM Symposium on High-Performance Parallel and Distributed Computing (HPDC) 2014 Message-ID: <52A379A2.9090901@cs.iit.edu> **** CALL FOR PAPERS **** The 23nd International ACM Symposium on High-Performance Parallel and Distributed Computing (HPDC-2014) Vancouver, Canada - June 23-27, 2014 http://www.hpdc.org/2014 The ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) is the premier annual conference for presenting the latest research on the design, implementation, evaluation, and application of parallel and distributed systems for high-end computing. In 2014, the 23nd HPDC and affiliated workshops will take place in the beautiful city of Vancouver, Canada during June 23-27, 2014. **** IMPORTANT DATES **** Abstracts Due: January 20, 2014 Papers Due: January 27, 2014 (no extensions) Author Rebuttal: March 14-18, 2014 Author Notifications: March 28, 2014 **** SCOPE AND TOPICS **** Submissions are welcomed on high-performance parallel and distributed computing topics including but not limited to: clouds, clusters, grids, Big Data, massively multicore, and global-scale computing systems. New scholarly research showing empirical and reproducible results in architectures, systems, and networks is strongly encouraged, as are experience reports of operational deployments that can provide insights for future research on HPDC applications and systems. All papers will be evaluated for their originality, technical depth and correctness, potential impact, relevance to the conference, and quality of presentation. Research papers must clearly demonstrate research contributions and novelty, while experience reports must clearly describe lessons learned and demonstrate impact. In the context of high-performance parallel and distributed computing, the topics of interest include, but are not limited to: o Systems, networks, and architectures for high-end computing o Massively multicore systems o Resource virtualization o Programming languages and environments o I/O, storage systems, and data management o Resource management and scheduling, including energy-aware techniques o Performance modeling and analysis o Fault tolerance, reliability, and availability o Data-intensive computing o Applications of parallel and distributed computing **** PAPER SUBMISSION GUIDELINES **** Authors are invited to submit technical papers of at most 12 pages in PDF format, including figures and references. Papers should be formatted in the ACM Proceedings Style (http://www.acm.org/sigs/publications/proceedings-templates) and submitted via the conference web site. No changes to the margins, spacing, or font sizes as specified by the style file are allowed. Accepted papers will appear in the conference proceedings, and will be incorporated into the ACM Digital Library. A limited number of papers will be accepted as posters. Papers must be self-contained and provide the technical substance required for the program committee to evaluate their contributions. Papers should thoughtfully address all related work, particularly work presented at previous HPDC events. Submitted papers must be original work that has not appeared in and is not under consideration for another conference or a journal. See the ACM Prior Publication Policy (http://www.acm.org/publications/policies/sim_submissions) for more details. **** HPDC'14 GENERAL CO-CHAIRS **** Beth Plale, Indiana University Matei Ripeanu, University of British Columbia **** HPDC'14 PROGRAM CO-CHAIRS **** Franck Cappello, Argonne National Lab and INRIA Dongyan Xu, Purdue University **** HPDC'14 PUBLICITY CHAIR **** Ioan Raicu, Illinois Institute of Technology, USA **** HPDC'14 PROGRAM COMMITTEE **** David Abramson, University of Queensland, Australia Gabriel Antoniu, INRIA, France Henri Bal, Vrije Universiteit, the Netherlands Adam Barker, University of St Andrews, UK Michela Becchi, University of Missouri - Columbia, USA Greg Bronevetsky, Lawrence Livermore National Laboratory, USA Ali Butt, Virginia Tech, USA Abhishek Chandra, University of Minnesota, USA Andrew Chien, University of Chicago and Argonne National Laboratory, USA Paolo Costa, Microsoft Research Cambridge, UK Dilma Da Silva, Qualcomm Research Silicon Valley, USA Peter Dinda, Northwestern University, USA Dick Epema, Delft and Eindhoven University of Technology, the Netherlands Gilles Fedak, INRIA, France Wu-chun Feng, Virginia Tech, USA Renato Figueiredo, University of Florida, USA Ian Foster, University of Chicago and Argonne National Laboratory, USA Kartik Gopalan, Binghamton University, USA Haryadi Gunawi, University of Chicago, USA Bingsheng He, Nanyang Technological University, Singapore Alexandru Iosup, Delft University of Technology, the Netherlands Shantenu Jha, Rutgers University, USA Hai Jin Huazhong, University of Science and Technology, China Larry Kaplan, Cray, Inc., USA Kate Keahey, Argonne National Laboratory, USA Thilo Kielmann, Vrije Universiteit, the Netherlands Charles Killian, Google, USA Zhiling Lan, Illinois Institute of Technology, USA John (Jack) Lange, University of Pittsburgh, USA Minglu Li, Shanghai Jiaotong University, China Barney Maccabe, Oak Ridge National Laboratory, USA Naoya Maruyama, RIKEN Advanced Institute for Computational Science, Japan Satoshi Matsuoka, Tokyo Institute of Technology, Japan Dejan Milojicic, HP Labs, USA Bogdan Nicolae, IBM Research, Ireland Sangmi Pallickara, Colorado State University, USA Manish Parashar, Rutgers University, USA Ioan Raicu, Illinois Institute of Technology, USA Prasenjit Sarkar, IBM Research, USA Daniele Scarpazza, D.E. Shaw Research, USA Karsten Schwan, Georgia Institute of Technology, USA Martin Swany, Indiana University, USA Michela Taufer, University of Delaware, USA Kenjiro Taura, University of Tokyo, Japan Douglas Thain, University of Notre Dame, USA Bhuvan Urgaonkar, Pennsylvania State University, USA Mustafa Uysal, VMware, Inc., USA Jon Weissman, University of Minnesota, USA Ming Zhao, Florida International University, USA **** HPDC STEERING COMMITTEE **** Henri Bal, Vrije Universiteit Andrew A. Chien, University of Chicago Peter Dinda, Northwestern University Dick Epema, Delft University of Technology Renato Figueiredo, University of Florida Ian Foster, Univ. of Chicago and Argonne National Laboratory Salim Hariri, University of Arizona Thilo Kielmann, Vrije Universiteit Amsterdam Arthur "Barney" Maccabe, Oak Ridge National Laboratory Manish Parashar, Rutgers University Matei Ripeanu, University of British Columbia Karsten Schwan, Georgia Tech Doug Thain, University of Notre Dame Jon Weissman, University of Minnesota (Chair) -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Editor: IEEE TCC, Springer Cluster, Springer JoCCASA Chair: IEEE/ACM MTAGS, ACM ScienceCloud ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ LinkedIn: http://www.linkedin.com/in/ioanraicu Google: http://scholar.google.com/citations?user=jE73HYAAAAAJ ================================================================= ================================================================= From iraicu at cs.iit.edu Sat Dec 7 13:47:11 2013 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Sat, 07 Dec 2013 13:47:11 -0600 Subject: [Swift-devel] Call for Workshops: ACM HPDC 2014 -- Proposals due December 9th, 2013 Message-ID: <52A37B3F.1010004@cs.iit.edu> The organizers of the /23rd International ACM Symposium on High-Performance Parallel and Distributed Computing/ (HPDC'14) *call for proposals for workshops* to be held with HPDC'14. The workshops will be held on June 23-27, 2014. Workshops should provide forums for discussion among researchers and practitioners on focused topics or emerging research areas relevant to the HPDC community. Organizers may structure workshops as they see fit, including invited talks, panel discussions, presentations of work in progress, fully peer-reviewed papers, or some combination. Workshops could be scheduled for half a day or a full day, depending on interest, space constraints, and organizer preference. Organizers should design workshops for approximately 20-40 participants, to balance impact and effective discussion. *Workshop proposal* will be uploaded in PDF format to the submission site by the deadline, and should include: * The name and acronym of the workshop * A description (0.5-1 page) of the theme of the workshop * A description (one paragraph) of the relation between the theme of the workshop and of HPDC * A list of topics of interest * The names and affiliations of the workshop organizers, and if applicable, of a significant portion of the program committee * A description of the expected structure of the workshop (papers, invited talks, panel discussions, etc.) * Data about previous offerings of the workshop (if any), including the attendance, the numbers of papers or presentations submitted and accepted, and the links to the corresponding websites * A publicity plan for attracting submissions and attendees. Please also include expected number of submissions, accepted papers, and attendees that you anticipate for a successful workshop. Due to publication deadlines, workshops must operate within roughly the following timeline: papers due mid February (2-3 weeks after the HPDC deadline), and selected and sent to the publisher by mid April. Workshop Chair: Martin Swany, Indiana University, USA * Questions: * For any questions regarding your submission, please contact the workshop chair at: hpdc-workshop-l [at] list [dot] indiana [dot] edu Important dates: *Workshop Proposals Due: * *December 9, 2013* Notifications: December 20, 2014 Workshop CFPs Online and Distributed: January 13, 2014 -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Editor: IEEE TCC, Springer Cluster, Springer JoCCASA Chair: IEEE/ACM MTAGS, ACM ScienceCloud ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ LinkedIn: http://www.linkedin.com/in/ioanraicu Google: http://scholar.google.com/citations?user=jE73HYAAAAAJ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Tue Dec 10 16:07:20 2013 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 10 Dec 2013 14:07:20 -0800 Subject: [Swift-devel] some details about softimage Message-ID: <1386713240.11830.3.camel@echo> Mike asked for some details of how that works. You specify it like this in sites.xml: /lustre/beagle/hategan/softimg.tar.gz /tmp/softimage That is . If your tar.gz contains a "start" executable in the root directory, it will be executed after unpacking. Similarly, if the tar.gz contains a "stop" executable in the root directory, it will be executed before cleanup (which is when no more jobs will be run on that node). Mihael From davidk at ci.uchicago.edu Wed Dec 11 12:55:37 2013 From: davidk at ci.uchicago.edu (David Kelly) Date: Wed, 11 Dec 2013 12:55:37 -0600 Subject: [Swift-devel] Question about app wildcards in trunk Message-ID: Hello, Quick question about the app wildcard feature in trunk/0.95. Suppose I have two sites defined, host1 and host2. I have an application called myapp. myapp is only installed on host1. Host2 does not have myapp. We're using this app definition file: * * * INSTALLED INTEL32::LINUX null host1 myapp /path/to/myapp In this config, myapp will run on both host1 and host2. When it tries to run on host2 it will fail because it's not in $PATH. Is it possible to say "only run myapp on host1, but use wildcards for every other app"? Thanks, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Wed Dec 11 14:17:46 2013 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 11 Dec 2013 12:17:46 -0800 Subject: [Swift-devel] Question about app wildcards in trunk In-Reply-To: References: Message-ID: <1386793066.19564.2.camel@echo> On Wed, 2013-12-11 at 12:55 -0600, David Kelly wrote: > Hello, > > Quick question about the app wildcard feature in trunk/0.95. > > Suppose I have two sites defined, host1 and host2. I have an application > called myapp. myapp is only installed on host1. Host2 does not have myapp. > We're using this app definition file: > > * * * INSTALLED INTEL32::LINUX null > host1 myapp /path/to/myapp > > In this config, myapp will run on both host1 and host2. When it tries to > run on host2 it will fail because it's not in $PATH. > > Is it possible to say "only run myapp on host1, but use wildcards for every > other app"? Unfortunately, not at this time. That was an issue I thought about, but couldn't see any elegant way of expressing that. I welcome discussions on how this would work in a backwards-compatible way. Or maybe it's time to drop the backwards compatible way. Mihael From wilde at mcs.anl.gov Thu Dec 12 16:03:42 2013 From: wilde at mcs.anl.gov (Michael Wilde) Date: Thu, 12 Dec 2013 16:03:42 -0600 (CST) Subject: [Swift-devel] Fwd: [Users] Appropriate use of $HOME In-Reply-To: <52AA13AD.7020808@lbl.gov> Message-ID: <2018060743.37536878.1386885822819.JavaMail.root@mcs.anl.gov> This note is very relevant to Swift... We need to show, in the Users Guide, good practices for filesystem usage. - Mike ----- Forwarded Message ----- From: "David Turner" To: users at nersc.gov Sent: Thursday, December 12, 2013 1:51:09 PM Subject: [Users] Appropriate use of $HOME Dear NERSC User, We have continued to receive reports of sluggish response for common interactive operations such as ls, df, vi, and others. Occasionally, user logins also take longer than expected. The purpose of this email to suggest how all users can help us solve these issues. Investigations consistently show two primary causes for this performance degradation: batch jobs performing significant I/O in global home directories, and parallel file transfer activities in global homes. NERSC's various file systems are tuned for different use cases. The home file systems are tuned for efficient small-file and metadata operations, as are typically encountered when building applications. Conversely, the various scratch and project file systems are tuned for high-volume streaming I/O operations. Please do not run batch jobs in your home directory. Instead, run batch jobs only from a scratch directory or a project directory. Also, even if running in a scratch or a project directory, try to avoid any explicit I/O to/from your home directory. Similarly, if you need to transfer data into or out of your home directory, please use no more than two simultaneous data transfer clients. This applies to multiple instances of ftp, hsi, htar, and cp and/or scp. Note that running multiple instances of these tools can not only put a heavy load on the file system, they can also put a heavy load on the login nodes themselves, contributing to sluggish interactive response. Lastly, file system performance can suffer when even a few active users are close to their disk quota limits. Everyone can benefit if you maintain sufficient "headroom" between your usage in $HOME and your 40GB quota. You can use the "myquota" command to view your disk usage and quota values. Effective use of HPSS can help you manage your data storage needs. If you have any questions about these issues, please contact NERSC Consulting at: 1-800-666-3772, menu option 3, 8 am - 5 pm, Pacific time (510) 486-8600, menu option 3, 8 am - 5 pm, Pacific time consult at nersc.gov http://help.nersc.gov/ Thank you for your cooperation in helping make NERSC compute and data systems responsive for all users. -- Best regards, David Turner User Services Group email: dpturner at lbl.gov NERSC Division phone: (510) 486-4027 Lawrence Berkeley Lab fax: (510) 486-4316 _______________________________________________ Users mailing list Users at nersc.gov From wilde at mcs.anl.gov Thu Dec 12 22:21:27 2013 From: wilde at mcs.anl.gov (Michael Wilde) Date: Thu, 12 Dec 2013 22:21:27 -0600 (CST) Subject: [Swift-devel] Try the "TrySwift" pre-release snapshot In-Reply-To: <1282366567.37713990.1386905217475.JavaMail.root@mcs.anl.gov> Message-ID: <553877591.37725532.1386908487077.JavaMail.root@mcs.anl.gov> Dear All, David Kelly of the Swift team has developed a web interface that lets you sample the Swift language via a web interface: http://swift-lang.org/tryswift You paste in and run Swift scripts; apps launched by your scripts are run on a few sandbox VMs. Swift tutorial text is presented in a side-bar for you to try. The goal is to let visitors to the Swift web quickly explore the language with no installation or setup. Its not quite ready to advertise and link to it from the Swift web, but we'd love to get your early feedback on this pre-release snapshot of the interface. Eventually, much of the Swift tutorial will be available to try from the companion frames of the interface. Please send comments and ideas for improvements back to swift-devel at ci.uchicago.edu. Thanks, and happy swifting! - Mike, for the Swift Team -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory From wilde at mcs.anl.gov Thu Dec 12 22:31:34 2013 From: wilde at mcs.anl.gov (Michael Wilde) Date: Thu, 12 Dec 2013 22:31:34 -0600 (CST) Subject: [Swift-devel] Try the "TrySwift" pre-release snapshot In-Reply-To: <553877591.37725532.1386908487077.JavaMail.root@mcs.anl.gov> Message-ID: <335216971.37725634.1386909094656.JavaMail.root@mcs.anl.gov> Swift Team: I bcc'ed this message to a large set of people with some connection to the project, but most of whom are not on swift-devel. Please keep an eye out for responses coming to swift-devel for Moderator approval, and let these through to the list so we can gather actionable feedback. Thanks, - Mike ----- Original Message ----- > From: "Michael Wilde" > To: "Swift Devel" > Cc: "Swift Devel" > Sent: Thursday, December 12, 2013 10:21:27 PM > Subject: [Swift-devel] Try the "TrySwift" pre-release snapshot > > Dear All, > > David Kelly of the Swift team has developed a web interface that lets > you sample the Swift language via a web interface: > > http://swift-lang.org/tryswift > > You paste in and run Swift scripts; apps launched by your scripts are > run on a few sandbox VMs. Swift tutorial text is presented in a > side-bar for you to try. > > The goal is to let visitors to the Swift web quickly explore the > language with no installation or setup. > > Its not quite ready to advertise and link to it from the Swift web, > but we'd love to get your early feedback on this pre-release > snapshot of the interface. Eventually, much of the Swift tutorial > will be available to try from the companion frames of the interface. > > Please send comments and ideas for improvements back to > swift-devel at ci.uchicago.edu. > > Thanks, and happy swifting! > > - Mike, for the Swift Team > > -- > Michael Wilde > Computation Institute, University of Chicago > Mathematics and Computer Science Division > Argonne National Laboratory > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > From wilde at mcs.anl.gov Thu Dec 12 22:44:18 2013 From: wilde at mcs.anl.gov (Michael Wilde) Date: Thu, 12 Dec 2013 22:44:18 -0600 (CST) Subject: [Swift-devel] Try the "TrySwift" pre-release snapshot In-Reply-To: <335216971.37725634.1386909094656.JavaMail.root@mcs.anl.gov> Message-ID: <580246805.37727041.1386909858104.JavaMail.root@mcs.anl.gov> > > Please send comments and ideas for improvements back to > > swift-devel at ci.uchicago.edu. 1. When you click on an output file, make the window that pops up much smaller, not much bigger than whats needed to view the output. 2. Show trace/tracef in an early example, ideally in Hello World. Add Swift synonyms print/printf for trace/tracef; add %d to tracef. 3. Provide a feedback box on the tryswift window. 4. Provide links to the User Guide and Tutorials list. 5. Provide a list of available app() defs and app entries. From wilde at mcs.anl.gov Thu Dec 12 22:58:08 2013 From: wilde at mcs.anl.gov (Michael Wilde) Date: Thu, 12 Dec 2013 22:58:08 -0600 (CST) Subject: [Swift-devel] Try the "TrySwift" pre-release snapshot In-Reply-To: <580246805.37727041.1386909858104.JavaMail.root@mcs.anl.gov> Message-ID: <767501952.37740445.1386910688934.JavaMail.root@mcs.anl.gov> 1. When you hit "Execute" on the initial window (containing just a comment, no executable statements) Swift hangs :( and the user never gets a response. Lets at least add this to the initial text: tracef("Hello, Swift World!\n"); (and fix that bug...) 2. Can we make it respond faster to trivial programs? Maybe we need to move the swift command itself to a non-VM, or perhaps can tune up the swift command a bit? ----- Original Message ----- > From: "Michael Wilde" > To: "Swift Devel" > Sent: Thursday, December 12, 2013 10:44:18 PM > Subject: Re: [Swift-devel] Try the "TrySwift" pre-release snapshot > > > > > Please send comments and ideas for improvements back to > > > swift-devel at ci.uchicago.edu. > > 1. When you click on an output file, make the window that pops up > much smaller, not much bigger than whats needed to view the output. > > 2. Show trace/tracef in an early example, ideally in Hello World. > Add Swift synonyms print/printf for trace/tracef; add %d to tracef. > > 3. Provide a feedback box on the tryswift window. > > 4. Provide links to the User Guide and Tutorials list. > > 5. Provide a list of available app() defs and app entries. > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > From wilde at mcs.anl.gov Thu Dec 12 23:04:32 2013 From: wilde at mcs.anl.gov (Michael Wilde) Date: Thu, 12 Dec 2013 23:04:32 -0600 (CST) Subject: [Swift-devel] Try the "TrySwift" pre-release snapshot In-Reply-To: <767501952.37740445.1386910688934.JavaMail.root@mcs.anl.gov> Message-ID: <1037854520.37748508.1386911072824.JavaMail.root@mcs.anl.gov> > 1. When you hit "Execute" on the initial window (containing just a > comment, no executable statements) Swift hangs :( and the user never > gets a response. Maybe its not hung. Maybe is just that "the user never gets a response". Lets bracket the swift command output (whether normally or with errors), something like this: --- swift command executing... Swift trunk swift-r7139 cog-r3806 RunID: 20131212-2258-4ke1dhcb Could not start execution: Compile error in procedure invocation at line 9: No function or procedure 'echo_app' found. swift command completed. (error above was intentionally created...) From hategan at mcs.anl.gov Fri Dec 13 04:28:59 2013 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Fri, 13 Dec 2013 02:28:59 -0800 Subject: [Swift-devel] Try the "TrySwift" pre-release snapshot In-Reply-To: <767501952.37740445.1386910688934.JavaMail.root@mcs.anl.gov> References: <767501952.37740445.1386910688934.JavaMail.root@mcs.anl.gov> Message-ID: <1386930539.10066.1.camel@echo> On Thu, 2013-12-12 at 22:58 -0600, Michael Wilde wrote: > 1. When you hit "Execute" on the initial window (containing just a comment, no executable statements) Swift hangs :( and the user never gets a response. > > Lets at least add this to the initial text: > > tracef("Hello, Swift World!\n"); > > (and fix that bug...) > > 2. Can we make it respond faster to trivial programs? Maybe we need to move the swift command itself to a non-VM, or perhaps can tune up the swift command a bit? Some I2u2 like thing could work, in which there is only one instance of swift loaded in memory and it runs everything. Not sure how this will fit here though. Mihael From foster at anl.gov Fri Dec 13 07:36:12 2013 From: foster at anl.gov (Ian Foster) Date: Fri, 13 Dec 2013 07:36:12 -0600 Subject: [Swift-devel] Try the "TrySwift" pre-release snapshot In-Reply-To: <553877591.37725532.1386908487077.JavaMail.root@mcs.anl.gov> References: <553877591.37725532.1386908487077.JavaMail.root@mcs.anl.gov> Message-ID: <9EC876B6-063F-4E17-B1AB-689808499BBC@anl.gov> Mike and David: This is really nice. A few comments and questions: -- It would be nice to be able to show the C code for "simulate" (for example) so that people understand the complete example. -- It would be interesting longer term to allow people to access data produced in earlier computations in later computations (?) -- Include the Swift logo somewhere -- Where are the VMs? -- Link to CI Connect? Ian. On Dec 12, 2013, at 10:21 PM, Michael Wilde wrote: > Dear All, > > David Kelly of the Swift team has developed a web interface that lets you sample the Swift language via a web interface: > > http://swift-lang.org/tryswift > > You paste in and run Swift scripts; apps launched by your scripts are run on a few sandbox VMs. Swift tutorial text is presented in a side-bar for you to try. > > The goal is to let visitors to the Swift web quickly explore the language with no installation or setup. > > Its not quite ready to advertise and link to it from the Swift web, but we'd love to get your early feedback on this pre-release snapshot of the interface. Eventually, much of the Swift tutorial will be available to try from the companion frames of the interface. > > Please send comments and ideas for improvements back to swift-devel at ci.uchicago.edu. > > Thanks, and happy swifting! > > - Mike, for the Swift Team > > -- > Michael Wilde > Computation Institute, University of Chicago > Mathematics and Computer Science Division > Argonne National Laboratory > From wilde at mcs.anl.gov Fri Dec 13 09:55:29 2013 From: wilde at mcs.anl.gov (Michael Wilde) Date: Fri, 13 Dec 2013 09:55:29 -0600 (CST) Subject: [Swift-devel] Try the "TrySwift" pre-release snapshot In-Reply-To: <9EC876B6-063F-4E17-B1AB-689808499BBC@anl.gov> Message-ID: <1223766167.37886485.1386950129409.JavaMail.root@mcs.anl.gov> Thanks, Ian - excellent ideas. Notes below > -- It would be nice to be able to show the C code for "simulate" (for > example) so that people understand the complete example. +1. (its a simple shell script at the moment; we'd like to recode in C, Python, R etc just to show the flexibility in a prospective user's "native" language. > -- It would be interesting longer term to allow people to access data > produced in earlier computations in later computations (?) +2. We'll need to associate a file "workspace" with your session. And put limiters on what can be created, perhaps via directory quotas and file name restrictions. Initial we can just keep it simple and trust users. Its just a VM after all. > -- Include the Swift logo somewhere +1 > -- Where are the VMs? At the moment, they are CI VMs. > -- Link to CI Connect? Plans for this are underway (also OSG Connect) From marialemos72 at gmail.com Fri Dec 13 19:11:38 2013 From: marialemos72 at gmail.com (WorldCIST) Date: Sat, 14 Dec 2013 01:11:38 +0000 Subject: [Swift-devel] Last Call - 2sd World Conference on IST; Submission: December 29 Message-ID: <20131214011151.174457CC0A4@mailrelay.anl.gov> ============================================================================ 2sd World Conference on Information Systems and Technologies - WorldCIST'14 April 15-18, 2014, Madeira Island, Portugal http://www.aisti.eu/worldcist14/ ================================ Submission deadline: December 29 * Proceedings published by Springer. ** Papers submitted for indexation by ISI, SCOPUS, DBLP, etc. *** Extended versions of best papers published in ISI/SCI/SSCI/JCR journals. Regards, Maria Lemos WorldCIST'14 http://www.aisti.eu/worldcist14/ From yadudoc1729 at gmail.com Fri Dec 13 20:14:29 2013 From: yadudoc1729 at gmail.com (Yadu Nand) Date: Fri, 13 Dec 2013 20:14:29 -0600 Subject: [Swift-devel] Knowledge lab app on Midway using new swift-config Message-ID: Hi, I tried the new swift-config on Swift-0.95 RC1 on Midway today for a text processing app from the Knowledge lab. Here are some of the things I noted. 1. Config is super easy! This is all I had to do : site=westmere site.westmere.slurm.exclusive=false use.provider.staging=true # App definition app.westmere.python=/usr/bin/python 2. Some settings were off when using my Swift-0.95 build. I had to module load swift and swift-conf, to get echo to be recognized as an app. Python still would throw the error "application "python" is not available for any site/pool in your tc.data catalog". This error should be corrected to point at swift.properties instead. 3. The .rlog file shows up the user directory. Can we have this to go to the run* dir. Occasionally .d folders also show up. 4. Why does a swift.log file show up in the user dir? It contains no useful info. Thanks, Yadu -------------- next part -------------- An HTML attachment was scrubbed... URL: From ketan at mcs.anl.gov Fri Dec 13 20:47:29 2013 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Fri, 13 Dec 2013 20:47:29 -0600 Subject: [Swift-devel] Knowledge lab app on Midway using new swift-config In-Reply-To: References: Message-ID: Cool. While we are at it, it might be of value to replace the use.provider.staging=true with a simpler term. May be explicit.staging or swift.staging .. On Fri, Dec 13, 2013 at 8:14 PM, Yadu Nand wrote: > Hi, > > I tried the new swift-config on Swift-0.95 RC1 on Midway today for a text > processing app from the Knowledge lab. > > Here are some of the things I noted. > 1. Config is super easy! This is all I had to do : > site=westmere > site.westmere.slurm.exclusive=false > use.provider.staging=true > # App definition > app.westmere.python=/usr/bin/python > > 2. Some settings were off when using my Swift-0.95 build. I had to module > load swift and swift-conf, to get echo to be recognized as an app. Python > still would throw the error "application "python" is not available for any > site/pool in your tc.data catalog". This error should be corrected to point > at swift.properties instead. > > 3. The .rlog file shows up the user directory. Can we have this to go to > the run* dir. > Occasionally .d folders also show up. > > 4. Why does a swift.log file show up in the user dir? It contains no > useful info. > > Thanks, > Yadu > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidk at ci.uchicago.edu Fri Dec 13 21:44:27 2013 From: davidk at ci.uchicago.edu (David Kelly) Date: Fri, 13 Dec 2013 21:44:27 -0600 Subject: [Swift-devel] Knowledge lab app on Midway using new swift-config In-Reply-To: References: Message-ID: Hey Yadu, Glad you were able to give it a try and get things working. Just a few notes: 2. Some settings were off when using my Swift-0.95 build. I had to module > load swift and swift-conf, to get echo to be recognized as an app. Python > still would throw the error "application "python" is not available for any > site/pool in your tc.data catalog". This error should be corrected to point > at swift.properties instead. > With Swift 0.95, swift.properties files get read in the following order: 1. The etc/swift.properties file included with the Swift distribution 2. $SWIFT_SITE_CONF/swift.properties - used for defining site templates. 3. $HOME/.swift/swift.properties 4. swift.properties in your current directory 5. Any property file you point to with the command line argument "-properties " As the swift.properties files get read, new settings override previous settings. When you load the swift 0.95RC1 module on Midway, $SWIFT_SITE_CONF gets set which contains all the information about midway. If you wanted to avoid loading the module, you could put the full definition of westmere in your local swift.properties. I think there's an argument to be made for including site definitions for all of the sites we support in the standard Swift package, rather than relying on them being loaded in a module. The problem you were running into with apps not being found is a misleading bug with the new config. If you say: site=westmere site.westmere.slurm.exclusive=false But you don't have the rest of westmere defined elsewhere in your properties path, you'll get only a partial sites.xml that doesn't have things like jobmanager defined. For some reason this triggers the app not found error. The fix is probably to add a check to make sure a site has certain mandatory properties defined or throw an error (jobmanager and workdir at least?). > 3. The .rlog file shows up the user directory. Can we have this to go to > the run* dir. > Occasionally .d folders also show up. > The rlog and *.d folder issues have been fixed in svn (after RC1). > 4. Why does a swift.log file show up in the user dir? It contains no > useful info. > The swift.log isn't created anymore with 0.95 or trunk (hooray!). That may have been generated when running with 0.94? Thanks for testing it out, please keep the feedback coming! :-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From wozniak at mcs.anl.gov Sat Dec 14 10:24:46 2013 From: wozniak at mcs.anl.gov (Justin M Wozniak) Date: Sat, 14 Dec 2013 09:24:46 -0700 Subject: [Swift-devel] Knowledge lab app on Midway using new swift-config In-Reply-To: References:

Message-ID: <52AC864E.50107@mcs.anl.gov> I think it's best to make properties hierarchical, like data.staging.mode=provider On 12/13/2013 07:47 PM, Ketan Maheshwari wrote: > Cool. While we are at it, it might be of value to replace the > use.provider.staging=true with a simpler term. May be explicit.staging > or swift.staging .. > > > On Fri, Dec 13, 2013 at 8:14 PM, Yadu Nand > wrote: > > Hi, > > I tried the new swift-config on Swift-0.95 RC1 on Midway today for > a text processing app from the Knowledge lab. > > Here are some of the things I noted. > 1. Config is super easy! This is all I had to do : > site=westmere > site.westmere.slurm.exclusive=false > use.provider.staging=true > # App definition > app.westmere.python=/usr/bin/python > > 2. Some settings were off when using my Swift-0.95 build. I had to > module load swift and swift-conf, to get echo to be recognized as > an app. Python still would throw the error "application "python" > is not available for any site/pool in your tc.data catalog". This > error should be corrected to point at swift.properties instead. > > 3. The .rlog file shows up the user directory. Can we have this to > go to the run* dir. > Occasionally .d folders also show up. > > 4. Why does a swift.log file show up in the user dir? It contains > no useful info. > > Thanks, > Yadu > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel -- Justin M Wozniak -------------- next part -------------- An HTML attachment was scrubbed... URL: From wilde at mcs.anl.gov Sat Dec 14 13:53:50 2013 From: wilde at mcs.anl.gov (Michael Wilde) Date: Sat, 14 Dec 2013 13:53:50 -0600 (CST) Subject: [Swift-devel] Knowledge lab app on Midway using new swift-config In-Reply-To: <52AC864E.50107@mcs.anl.gov> Message-ID: <1570525711.38367490.1387050830954.JavaMail.root@mcs.anl.gov> > I think it's best to make properties hierarchical, like > data.staging.mode=provider +1 We should all all the pre-0.95 swift.properties to the property renaming sheet and rename as needed, as was done for the sites.xml properties, before the next 0.95 RC. - Mike From davidk at ci.uchicago.edu Sat Dec 14 15:50:27 2013 From: davidk at ci.uchicago.edu (David Kelly) Date: Sat, 14 Dec 2013 15:50:27 -0600 Subject: [Swift-devel] Knowledge lab app on Midway using new swift-config In-Reply-To: <1570525711.38367490.1387050830954.JavaMail.root@mcs.anl.gov> References: <52AC864E.50107@mcs.anl.gov> <1570525711.38367490.1387050830954.JavaMail.root@mcs.anl.gov> Message-ID: I like data.staging.mode=provider and data.staging.mode=wrapper. I added the list of all documented (and one undocumented) swift properties to the google doc. The first table is related to coaster properties. The second table is general swift properties. https://docs.google.com/document/d/1NTqqwVOM7QKUelx-sIabhDdkzH1kiNSpT0S4qN_uTeY On Sat, Dec 14, 2013 at 1:53 PM, Michael Wilde wrote: > > > I think it's best to make properties hierarchical, like > > data.staging.mode=provider > > +1 > > We should all all the pre-0.95 swift.properties to the property renaming > sheet and rename as needed, as was done for the sites.xml properties, > before the next 0.95 RC. > > - Mike > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wilde at mcs.anl.gov Sat Dec 14 16:25:23 2013 From: wilde at mcs.anl.gov (Michael Wilde) Date: Sat, 14 Dec 2013 16:25:23 -0600 (CST) Subject: [Swift-devel] Knowledge lab app on Midway using new swift-config In-Reply-To: Message-ID: <1403595058.38371852.1387059923586.JavaMail.root@mcs.anl.gov> > I like data.staging.mode=provider and data.staging.mode=wrapper. and data.staging.mode={normal,default,???} > I added the list of all documented (and one undocumented) swift > properties > to the google doc. Great - thanks! Should we grep the code for more undocumented ones? Peripherally related: is it time to remove kickstart? - Mike From yadudoc1729 at gmail.com Sat Dec 14 19:58:08 2013 From: yadudoc1729 at gmail.com (Yadu Nand) Date: Sat, 14 Dec 2013 19:58:08 -0600 Subject: [Swift-devel] Knowledge lab app on Midway using new swift-config In-Reply-To: <1403595058.38371852.1387059923586.JavaMail.root@mcs.anl.gov> References: <1403595058.38371852.1387059923586.JavaMail.root@mcs.anl.gov> Message-ID: How do we handle cases where we need files moved using more than one staging mechanism ? How would the user say what mechanism to use for some type of files, and also what would the staging mode be set to ? data.staging.mode= looks good. David, Yes, I think we should have site-definitions in the swift builds in addition to site-specific modules. On Sat, Dec 14, 2013 at 4:25 PM, Michael Wilde wrote: > > > I like data.staging.mode=provider and data.staging.mode=wrapper. > and > data.staging.mode={normal,default,???} > > > I added the list of all documented (and one undocumented) swift > > properties > > to the google doc. > > Great - thanks! > > Should we grep the code for more undocumented ones? > > Peripherally related: is it time to remove kickstart? > > - Mike > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > -- Yadu Nand B -------------- next part -------------- An HTML attachment was scrubbed... URL: From yadudoc1729 at gmail.com Fri Dec 20 12:35:29 2013 From: yadudoc1729 at gmail.com (Yadu Nand) Date: Fri, 20 Dec 2013 12:35:29 -0600 Subject: [Swift-devel] Duplicate workdirectory entry in generated sites.xml Message-ID: Hi, I'm seeing a duplicate entry for workdirectory when using the new config mechanism ?in Swift 0.95. I think this entry is confusing the staging mechanism. Is there some workaround for this ? Here's the entry for local in swift.properties : site.local { provider=local:local jobsPerNode=1 maxWalltime=01:00:00 initialScore=10000 filesystem=local workdirectory=/scratch/midway/$USER/work jobthrottle=0.1999 } Here's the generated sites.xml: 0 1 1 01:00:00 /home/yadunand/knowledge_lab-tfidf_example/run007 /scratch/midway/yadunand/work 0.1999 10000 ? ?Thanks, ?Yadu -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidk at ci.uchicago.edu Fri Dec 20 13:08:20 2013 From: davidk at ci.uchicago.edu (David Kelly) Date: Fri, 20 Dec 2013 13:08:20 -0600 Subject: [Swift-devel] Duplicate workdirectory entry in generated sites.xml In-Reply-To: References: Message-ID: Hey Yadu, Local is defined in the default swift.properties and it uses the new naming configuration described in http://swiftlang.org/guides/trunk/designs/swiftrun.html. Try using workdir instead of workdirectory. On Fri, Dec 20, 2013 at 12:35 PM, Yadu Nand wrote: > Hi, > > I'm seeing a duplicate entry for workdirectory when using the new config > mechanism in Swift 0.95. I think this entry is confusing the staging > mechanism. > > Is there some workaround for this ? > > Here's the entry for local in swift.properties : > site.local { > provider=local:local > jobsPerNode=1 > maxWalltime=01:00:00 > initialScore=10000 > filesystem=local > workdirectory=/scratch/midway/$USER/work > jobthrottle=0.1999 > } > > Here's the generated sites.xml: > > > > 0 > 1 > 1 > 01:00:00 > > /home/yadunand/knowledge_lab-tfidf_example/run007 ctory> > /scratch/midway/yadunand/work > 0.1999 > 10000 > > > > Thanks, > Yadu > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cpsosa at cray.com Mon Dec 23 20:55:00 2013 From: cpsosa at cray.com (Carlos P. Sosa) Date: Tue, 24 Dec 2013 02:55:00 -0000 Subject: [Swift-devel] White papers Message-ID: <9AB7792F032F864D93FEF34EF398060424D630CC@CFWEX01.americas.cray.com> Hello Are there any white papers that describe swift? Thanks carlos -------------- next part -------------- An HTML attachment was scrubbed... URL: