From ajmazurie at oenone.net Wed Jun 10 14:36:13 2015 From: ajmazurie at oenone.net (=?utf-8?Q?Aur=C3=A9lien_Mazurie?=) Date: Wed, 10 Jun 2015 13:36:13 -0600 Subject: [Swift-user] Management of undeclared files Message-ID: <68C04C80-5609-4485-BCCB-F26366FD7F83@oenone.net> Dear all, After making my way through the Swift documentation I started writing test workflows to see how simple (then more complex) tasks would be managed by this platform. One such test workflow is giving me issues, and I couldn't find an obvious solution in the documentation. I wrote this script, http://pastebin.com/zQZQphCd to format a FASTA file using the NCBI BLAST utility makeblastdb. What is interesting is that although the Swift script run without error, I cannot find the files makeblastdb is meant to produce anywhere on my hard drive. I do get the correct log file (log_file variable), and this log shows that makeblastdb ran successfully. However I should also get three other files, .nin, .nhr and .nsq. My question is: where are these files? Since makeblastdb ran without issue, they probably have been created somewhere (and I confirm that they are created when I run the exact same command on the terminal). Are they automatically deleted by Swift since they are not explicitly named by another variable? I'm guessing that the answer is yes; if that the case, is there a way to have variables storing file names that are only known at execution time? For example, I would like my 'FormatLibrary' function to return among else the name of these three files, rather than having to bind these file names to variables before the execution of this function. Is that even doable? Best, Aur?lien Mazurie -------------- next part -------------- An HTML attachment was scrubbed... URL: From wilde at anl.gov Wed Jun 10 15:07:34 2015 From: wilde at anl.gov (Michael Wilde) Date: Wed, 10 Jun 2015 15:07:34 -0500 Subject: [Swift-user] Management of undeclared files In-Reply-To: <68C04C80-5609-4485-BCCB-F26366FD7F83@oenone.net> References: <68C04C80-5609-4485-BCCB-F26366FD7F83@oenone.net> Message-ID: <55789906.8000503@anl.gov> Dear Aur?lien, Your files were likely created in the task "sandbox" directory that Swift creates for each app invocation. This directory is below the workdirectory for the execution site (probably your local machine unless you configured Swift to run on other sites). The basic language model is that for each app you declare what files go into the app run, and which are returned (as files, not strings). There are many ways to do this: - declare the nin, nhr, and nsq files as outputs of your app (which you say below you'd rather avoid) - declare a struct of 3 variables (for this we need to see if an existing mapper would work, else you'd need to use a mapper script) - let swift name the output files for you (but then you have to tell MAKEBLASTDB what to name its files). The first approach is simplest (see example below) and quite easy once you get used to this model of scripting. I suspect we can suggest a more compact way with a bit more thinking. - Mike untested example: app (file log_file, file nin, file nhr, file nsq) FormatLibrary (file fasta_file, string library_name) { MAKEBLASTDB ...; } file input_fasta <"test_blast.fasta">; file log_file <"test_blast.makeblastdb.log">; string library_name = strcut(filename(input_fasta), "^(.*?)\\.fa(sta)?"); file nin > My question is: where are these files? Since makeblastdb ran without > issue, they probably have been created somewhere (and I confirm that > they are created when I run the exact same command on the terminal). > Are they automatically deleted by Swift since they are not explicitly > named by another variable? > > I'm guessing that the answer is yes; if that the case, is there a way > to have variables storing file names that are only known at execution > time? For example, I would like my 'FormatLibrary' function to return > among else the name of these three files, rather than having to bind > these file names to variables before the execution of this function. > Is that even doable? -- Michael Wilde Mathematics and Computer Science Computation Institute Argonne National Laboratory The University of Chicago From iraicu at cs.iit.edu Thu Jun 11 07:34:23 2015 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Thu, 11 Jun 2015 07:34:23 -0500 Subject: [Swift-user] CFP: 2nd IEEE/ACM International Symposium on Big Data Computing (BDC) 2015 -- co-located with IEEE/ACM UCC 2015 Message-ID: <5579804F.7010202@cs.iit.edu> CALL FOR PAPERS --------------------------------------------------------------------------------------- 2nd IEEE/ACM International Symposium on Big Data Computing (BDC) 2015 http://datasys.cs.iit.edu/events/BDC2015/ --------------------------------------------------------------------------------------- December 7-10, 2015 St. Raphael Resort, Limassol, Cyprus Co-located with 8th IEEE/ACM International Conference on Utility and Cloud Computing ======================================================================================= Rapid advances in digital sensors, networks, storage, and computation along with their availability at low cost is leading to the creation of huge collections of data -- dubbed as Big Data. This data has the potential for enabling new insights that can change the way business, science, and governments deliver services to their consumers and can impact society as a whole. This has led to the emergence of the Big Data Computing paradigm focusing on sensing, collection, storage, management and analysis of data from variety of sources to enable new value and insights. To realize the full potential of Big Data Computing, we need to address several challenges and develop suitable conceptual and technological solutions for dealing them. These include life-cycle management of data, large-scale storage, flexible processing infrastructure, data modeling, scalable machine learning and data analysis algorithms, techniques for sampling and making trade-off between data processing time and accuracy, and dealing with privacy and ethical issues involved in data sensing, storage, processing, and actions. The International Symposium on Big Data Computing (BDC) 2015 -- held in conjunction with 8th IEEE/ACM International Conference on Utility and Cloud Computing (UCC) 2015, December 7-10, 2015, St. Raphael Resort, Limassol, Cyprus, aims at bringing together international researchers, developers, policy makers, and users and to provide an international forum to present leading research activities, technical solutions, and results on a broad range of topics related to Big Data Computing paradigms, platforms and their applications. The conference features keynotes, technical presentations, posters, and workshops. TOPICS: --------------------------------------------------------------------------------------- I. Big Data Science * Analytics * Algorithms for Big Data * Energy-efficient Algorithms * Big Data Search * Big Data Acquisition, Integration, Cleaning, and Best Practices * Visualization of Big Data II. Big Data Infrastructures and Platforms * Programming Systems * Cyber-Infrastructure * Performance evaluation * Fault tolerance and reliability * I/O and Data management * Storage Systems (including file systems, NoSQL, and RDBMS) * Resource management * Many-Task Computing * Many-core computing and accelerators III. Big Data Security and Policy * Management Policies * Data Privacy * Data Security * Big Data Archival and Preservation * Big Data Provenance IV. Big Data Applications * Scientific application cases studies on Cloud infrastructure * Big Data Applications at Scale * Experience Papers with Big Data Application Deployments * Data streaming applications * Big Data in Social Networks * Healthcare Applications * Enterprise Applications IMPORTANT DATES --------------------------------------------------------------------------------------- Papers: * Papers Due: 03 July, 2015 * Author Notifications: 21 August, 2015 * Final Papers Due: 21 September, 2015 Posters * Proceedings-published posters due: 28 August, 2015 * Notification of acceptance: 18 September, 2015 * Camera-ready posters due: 21 September, 2015 PAPER SUBMISSION --------------------------------------------------------------------------------------- Authors are invited to submit papers electronically. Submitted manuscripts should be structured as technical papers and may not exceed 10 letter size (8.5 x 11) pages including figures, tables and references using the following templates (latex, pdf, doc). Authors should submit the manuscript in PDF format and make sure that the file will print on a printer that uses letter size (8.5 x 11) paper. The official language of the meeting is English. All manuscripts will be reviewed and will be judged on correctness, originality, technical strength, significance, quality of presentation, and interest and relevance to the conference attendees. Papers conforming to the above guidelines can be submitted through the BDC 2015 paper submission system (https://www.easychair.org/conferences/?conf=bdc2015). Submitted papers must represent original unpublished research that is not currently under review for any other conference or journal. Papers not following these guidelines will be rejected without review and further action may be taken, including (but not limited to) notifications sent to the heads of the institutions of the authors and sponsors of the conference. Submissions received after the due date, exceeding length limit, or not appropriately structured may also not be considered. Authors may contact the conference PC Chair for more information. Selected papers from BDC 2015 will be invited to extend and submit to the Special Issue on Big Data Computing in the IEEE Transaction on Cloud Computing. ORGANIZATION --------------------------------------------------------------------------------------- General Chairs * Rajkumar Buyya, University of Melbourne, Australia * George Angelos Papadopoulos,University of Cyprus, Cyprus Program Committee Chairs (bdc15-chairs at datasys.cs.iit.edu) * Ioan Raicu, Illinois Institute of Technology & Argonne National Laboratory, USA * Amy Apon, National Science Foundation, USA * Manish Parashar, Rutgers University, USA Program Committee Vice Chairs * Omer Rana, Cardiff University, UK * Ilkay Altintas, University of California, San Diego, USA Program Committee Members * Alexander Rasin, DePaul University, USA * Alok Choudhary, Northwestern University, USA * Abhishek Chandra, University of Minnesota, USA * Andre Luckow, BMW IT Research Center, USA * Daniel Katz, University of Chicago and Argonne National Lab, USA * Dongfang Zhao, Illinois Institute of Technology, USA * Douglas Thain, University of Notre Dame, USA * Florian Schintke, Zuse Institute Berlin, Germany * Giuliano Casale, Imperial College London, UK * Jaliya Ekanayake, Microsoft, USA * Jessica Chen-Burger, Heriot-Watt University, UK * Judy Qiu, Indiana University, USA * Justin Wozniak, Argonne National Lab, USA * Ke Wang, Illinois Institute of Technology, USA * Kesheng (John) Wu, Lawrence Berkeley National Lab, USA * Kyle Chard, University of Chicago and Argonne National Lab, USA * Lavanya Ramakrishnan, Lawrence Berkeley National Laboratory, USA * Marco Netto, IBM Research, Brazil * Matei Ripeanu, University of British Columbia, Canada * Matei Stroila, HERE, USA * Nagiza Samatova, North Carolina State University, USA * Paolo Missier, Newcastle University, UK * Paul Watson, NewCastle University, UK * Peter Burnap, Cardiff University, UK * Rahul Potharaju, Microsoft, USA * Rajkumar Kettimuthu, Argonne National Lab and University of Chicago, USA * Robert Ross, Argonne National Lab, USA * Samer Al-Kiswany, University of British Columbia, Canada * Scott Klasky, Oak Ridge National Lab, USA * Wei Tang, Argonne National Lab, USA * Weidong Shi, University of Houston, USA * Xiaolin (Andy) Li, University of Florida, USA * Yanlong Yin, Bloomberg, USA * Yong Chen, Texas Tech University, USA * Yong Zhao, University of Electronic Science and Technology, China * Zhao Zhang, University of California, Berkeley, USA Cyber Co-Chairs * Dongfang Zhao, Illinois Institute of Technology, USA Local Organizing Committee * George Angelos Papadopoulos, University of Cyprus, Cyprus SPONSORSHIP --------------------------------------------------------------------------------------- IEEE*, ACM*, and IIT* *Final approval on sponsorship is pending. -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Editor: IEEE TCC, Springer Cluster, Springer JoCCASA Chair: IEEE/ACM MTAGS, ACM ScienceCloud ================================================================= Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ LinkedIn: http://www.linkedin.com/in/ioanraicu Google: http://scholar.google.com/citations?user=jE73HYAAAAAJ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From iraicu at cs.iit.edu Wed Jun 17 01:04:49 2015 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Tue, 16 Jun 2015 23:04:49 -0700 Subject: [Swift-user] Call for Participation: IEEE Cluster 2015 in Chicago, September 8-11 Message-ID: <55810E01.1010008@cs.iit.edu> IEEE International Conference on Cluster Computing September 8-11, 2015 Chicago, IL, USA http://www.mcs.anl.gov/ieeecluster2015/ **** NEWS **** * 38 full papers accepted out of 157 submissions (24% acceptance rate) * 28 short papers with oral presentations * 30 peer-reviewed posters * 3 keynotes * 6 workshops * Panel on exascale computing * 2 industry sessions with talks from Cray and DDN * Student Mentoring Program * Student Travel Support * Early bird registration: July 25th, 2015 * Hotel Reservation: August 17th, 2015 **** CALL FOR PARTICIPATION **** Clusters have become the workhorse for computational science and engineering research, powering innovation and discovery that advance science and society. They are the base for building today?s rapidly evolving cloud and HPC infrastructures, and are used to solve some of the most complex problems. Clusters are the base for building today?s rapidly evolving cloud and HPC infrastructure, and are used to solve some of the most complex problems. The challenge to make them scalable, efficient, and more functional requires the common effort of the cluster community in the areas of cluster system design, management and monitoring ? at hardware, system, middleware, and application level. The IEEE Cluster conference has precisely this goal: to be a major venue for such collective effort. We are happy to announce a rich, high-quality conference program for IEEE Cluster 2015 with 38 full papers accepted out of 157 submissions that went through a selective reviewing process (over 600 reviews), resulting in an acceptance rate of 24%. In addition the conference features 28 short papers and 30 peer-reviewed posters present exciting, emergent research directions. The conference program includes three keynotes, six workshops, a panel on exascale computing, and two industry sessions with talks from Cray and Data Direct Networks (DDN) - two major players with high impact in the area of cluster computing. We continue the successful Student Mentoring Program specific to PhD students, which will include student travel support; for more information, see http://www.mcs.anl.gov/ieeecluster2015/student-program/. The entire program of IEEE Cluster is now available at http://press3.mcs.anl.gov/ieeecluster2015/conference-program/technical-program. The Organizing Committee is happy to welcome you to Chicago and hopes you find Cluster 2015 an exciting and intellectually stimulating event! *Venue and Accommodation* The conference and the affiliated workshops are held in the Double Tree Hotel Chicago Magnificent Mile, which is centrally located and within walking distance of numerous attractions in downtown Chicago, Illinois. The hotel is holding a block of rooms reserved at a very special rate for Cluster 2015 participants. Book your room at: http://www.mcs.anl.gov/ieeecluster2015/registration-and-accommodation/accommodation/ *Registration* Register for the conference at http://www.mcs.anl.gov/ieeecluster2015/registration-and-accommodation/registration/ * **Important Dates* Early Bird Conference Registration Cut-off Date: July 25, 2015 Hotel Reservation Cut-off: August 17, 2015 Main Conference: September 8-11, 2015 Workshops: September 8, 2015 -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Editor: IEEE TCC, Springer Cluster, Springer JoCCASA Chair: IEEE/ACM MTAGS, ACM ScienceCloud ================================================================= Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ LinkedIn: http://www.linkedin.com/in/ioanraicu Google: http://scholar.google.com/citations?user=jE73HYAAAAAJ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From maxhutch at gmail.com Wed Jun 17 15:49:38 2015 From: maxhutch at gmail.com (Max Hutchinson) Date: Wed, 17 Jun 2015 20:49:38 +0000 Subject: [Swift-user] iterate adding dependencies Message-ID: Hello, I'm using an iterate loop like this: progress[0] = 0.; step = 5; iterate i { // Build executable if (i == 0){ // Run from initial condition output[i] = run_new(); } else { // Resume from old run output[i] = run_old(output[i-1]); } // Expensive post-processing refined[i] = post_process(output[i]); progress[i+1] = progress[i] + step; } until (progress[i] < 100.); run_old doesn't depend on refined, but will not run until the post_process application completes (which is a while; it is expensive). I'm inferring that iterate is sequential at the iteration-granularity. Is this intentional? I think that iterate is the natural way to express this computation, but the serialization of run_old wrt post_process is very expensive. Is there a case where the assumption that iterations are evaluated sequentially changes the result of the program? Max -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Wed Jun 17 16:35:49 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 17 Jun 2015 14:35:49 -0700 Subject: [Swift-user] iterate adding dependencies In-Reply-To: References: Message-ID: <1434576949.9826.25.camel@echo3> Hi, There is no good reason why that would be the case. The only things that need to be serialized are the things that the "until" expression depends on. In other words the sequencing is only so that the number of iterations can be determined unambiguously. So the issue should be fixed. In the mean time, there are two workarounds I can think of: 1. Use foreach instead of iterate: progress[0] = 0.; step = 5; foreach p, i in progress { // Build executable if (i == 0){ // Run from initial condition output[i] = run_new(); } else { // Resume from old run output[i] = run_old(output[i-1]); } // Expensive post-processing refined[i] = post_process(output[i]); if (progress[i] < 100) { progress[i+1] = progress[i] + step; } } 2. Move post_process outside of iterate: progress[0] = 0.; step = 5; iterate i { // Build executable if (i == 0){ // Run from initial condition output[i] = run_new(); } else { // Resume from old run output[i] = run_old(output[i-1]); } progress[i+1] = progress[i] + step; } until (progress[i] < 100.); foreach v, i in output { // Expensive post-processing refined[i] = post_process(output[i]); } Mihael On Wed, 2015-06-17 at 20:49 +0000, Max Hutchinson wrote: > Hello, > > I'm using an iterate loop like this: > > progress[0] = 0.; > step = 5; > iterate i { > // Build executable > if (i == 0){ > // Run from initial condition > output[i] = run_new(); > } else { > // Resume from old run > output[i] = run_old(output[i-1]); > } > // Expensive post-processing > refined[i] = post_process(output[i]); > progress[i+1] = progress[i] + step; > } until (progress[i] < 100.); > > run_old doesn't depend on refined, but will not run until the post_process > application completes (which is a while; it is expensive). I'm inferring > that iterate is sequential at the iteration-granularity. > > Is this intentional? I think that iterate is the natural way to express > this computation, but the serialization of run_old wrt post_process is very > expensive. Is there a case where the assumption that iterations are > evaluated sequentially changes the result of the program? > > Max > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user From maxhutch at gmail.com Thu Jun 18 11:52:45 2015 From: maxhutch at gmail.com (Max Hutchinson) Date: Thu, 18 Jun 2015 16:52:45 +0000 Subject: [Swift-user] direct staging usage Message-ID: Hello, https://github.com/swift-lang/swift-k/blob/master/docs/staging/staging.asc and http://swift-lang.org/guides/trunk/userguide/userguide.html#_configuration indicate that "scratch" is optional, but when running with direct staging I get this error: missing required site property: 'scratch' Is 'scratch' mandatory for direct staging? Thanks, Max -------------- next part -------------- An HTML attachment was scrubbed... URL: From yadunand at uchicago.edu Thu Jun 18 11:56:47 2015 From: yadunand at uchicago.edu (Yadu Nand Babuji) Date: Thu, 18 Jun 2015 11:56:47 -0500 Subject: [Swift-user] direct staging usage In-Reply-To: References: Message-ID: <5582F84F.404@uchicago.edu> Hi Max, "scratch" should be optional. Could you please send me the swift.conf and a tar-ball of the runNNN directory from a failed run. Thanks, Yadu On 06/18/2015 11:52 AM, Max Hutchinson wrote: > Hello, > > https://github.com/swift-lang/swift-k/blob/master/docs/staging/staging.asc > and > http://swift-lang.org/guides/trunk/userguide/userguide.html#_configuration > indicate that "scratch" is optional, but when running with direct > staging I get this error: > missing required site property: 'scratch' > Is 'scratch' mandatory for direct staging? > > Thanks, > Max > > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From maxhutch at gmail.com Thu Jun 18 12:20:02 2015 From: maxhutch at gmail.com (Max Hutchinson) Date: Thu, 18 Jun 2015 17:20:02 +0000 Subject: [Swift-user] import position in swift script Message-ID: Do all import statements need to come before executable statements? I have two scripts: defs.swift and run.swift. If run.swift starts with import "defs"; then everything works fine. If instead defs.swift ends with import "run"; I get this error: Swift trunk git-rev: 1d512cbda070d363e8ec3cf16e77dfcddc6e66cb heads/master 6377 RunID: run011 Could not compile SwiftScript source: line 24:12: unexpected token: ; where line 24 is: int foo = 1; Thanks, Max -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Thu Jun 18 12:47:53 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 18 Jun 2015 10:47:53 -0700 Subject: [Swift-user] import position in swift script In-Reply-To: References: Message-ID: <1434649673.11316.0.camel@echo3> Hi, Yes. Imports must come before other types of statements. Mihael On Thu, 2015-06-18 at 17:20 +0000, Max Hutchinson wrote: > Do all import statements need to come before executable statements? > > I have two scripts: defs.swift and run.swift. If run.swift starts with > import "defs"; > then everything works fine. If instead defs.swift ends with > import "run"; > I get this error: > Swift trunk git-rev: 1d512cbda070d363e8ec3cf16e77dfcddc6e66cb heads/master > 6377 > RunID: run011 > Could not compile SwiftScript source: line 24:12: unexpected token: ; > where line 24 is: > int foo = 1; > > Thanks, > Max > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user From maxhutch at gmail.com Thu Jun 18 12:58:15 2015 From: maxhutch at gmail.com (Max Hutchinson) Date: Thu, 18 Jun 2015 17:58:15 +0000 Subject: [Swift-user] import position in swift script In-Reply-To: <1434649673.11316.0.camel@echo3> References: <1434649673.11316.0.camel@echo3> Message-ID: Could that constraint be relaxed? On Thu, Jun 18, 2015 at 12:47 PM Mihael Hategan wrote: > Hi, > > Yes. Imports must come before other types of statements. > > Mihael > > On Thu, 2015-06-18 at 17:20 +0000, Max Hutchinson wrote: > > Do all import statements need to come before executable statements? > > > > I have two scripts: defs.swift and run.swift. If run.swift starts with > > import "defs"; > > then everything works fine. If instead defs.swift ends with > > import "run"; > > I get this error: > > Swift trunk git-rev: 1d512cbda070d363e8ec3cf16e77dfcddc6e66cb > heads/master > > 6377 > > RunID: run011 > > Could not compile SwiftScript source: line 24:12: unexpected token: ; > > where line 24 is: > > int foo = 1; > > > > Thanks, > > Max > > _______________________________________________ > > Swift-user mailing list > > Swift-user at ci.uchicago.edu > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Thu Jun 18 13:07:08 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 18 Jun 2015 11:07:08 -0700 Subject: [Swift-user] import position in swift script In-Reply-To: References: <1434649673.11316.0.camel@echo3> Message-ID: <1434650828.11812.4.camel@echo3> Yes, but I don't think it should. Why is it important to have import statements scattered through the script? Mihael On Thu, 2015-06-18 at 17:58 +0000, Max Hutchinson wrote: > Could that constraint be relaxed? > > On Thu, Jun 18, 2015 at 12:47 PM Mihael Hategan wrote: > > > Hi, > > > > Yes. Imports must come before other types of statements. > > > > Mihael > > > > On Thu, 2015-06-18 at 17:20 +0000, Max Hutchinson wrote: > > > Do all import statements need to come before executable statements? > > > > > > I have two scripts: defs.swift and run.swift. If run.swift starts with > > > import "defs"; > > > then everything works fine. If instead defs.swift ends with > > > import "run"; > > > I get this error: > > > Swift trunk git-rev: 1d512cbda070d363e8ec3cf16e77dfcddc6e66cb > > heads/master > > > 6377 > > > RunID: run011 > > > Could not compile SwiftScript source: line 24:12: unexpected token: ; > > > where line 24 is: > > > int foo = 1; > > > > > > Thanks, > > > Max > > > _______________________________________________ > > > Swift-user mailing list > > > Swift-user at ci.uchicago.edu > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > > > > > From maxhutch at gmail.com Thu Jun 18 13:18:08 2015 From: maxhutch at gmail.com (Max Hutchinson) Date: Thu, 18 Jun 2015 18:18:08 +0000 Subject: [Swift-user] import position in swift script In-Reply-To: <1434650828.11812.4.camel@echo3> References: <1434649673.11316.0.camel@echo3> <1434650828.11812.4.camel@echo3> Message-ID: Not scattered; at the end. I will have ~10 use cases, each written as a different defs file (i.e. defs_a.swift, defs_b.swift, etc). For each use-case, the run.swift, which is a big foreach loop, is the same. Given this structure, I can think of at least 4 organizations: 1) run.swift imports "defs_a" or "defs_b". To change use cases, the import in run.swift is edited. This is what I currently do but forces me to either serialize inter-script execution or edit scripts that are in the process of being executed. 2) defs_a.swift, defs_b.swift, etc import "run". This seems clean and is what I want to do. 3) Wrapper scripts are created to contain both imports (i.e. a_wrap.swift : ```import "defs_a"; import "run"```). This currently works, but results in script bloat. 4) Wrap the contents of run.swift in a procedure, import it at the top of defs_{a,b,...}.swift, but then call it at the bottom. This is surely the "right" way to do it, but its more effort and could have argument list bloat. Since each of these option is semantically identical, why not allow the user to pick their preference? On Thu, Jun 18, 2015 at 1:07 PM Mihael Hategan wrote: > Yes, but I don't think it should. > > Why is it important to have import statements scattered through the > script? > > Mihael > > On Thu, 2015-06-18 at 17:58 +0000, Max Hutchinson wrote: > > Could that constraint be relaxed? > > > > On Thu, Jun 18, 2015 at 12:47 PM Mihael Hategan > wrote: > > > > > Hi, > > > > > > Yes. Imports must come before other types of statements. > > > > > > Mihael > > > > > > On Thu, 2015-06-18 at 17:20 +0000, Max Hutchinson wrote: > > > > Do all import statements need to come before executable statements? > > > > > > > > I have two scripts: defs.swift and run.swift. If run.swift starts > with > > > > import "defs"; > > > > then everything works fine. If instead defs.swift ends with > > > > import "run"; > > > > I get this error: > > > > Swift trunk git-rev: 1d512cbda070d363e8ec3cf16e77dfcddc6e66cb > > > heads/master > > > > 6377 > > > > RunID: run011 > > > > Could not compile SwiftScript source: line 24:12: unexpected token: ; > > > > where line 24 is: > > > > int foo = 1; > > > > > > > > Thanks, > > > > Max > > > > _______________________________________________ > > > > Swift-user mailing list > > > > Swift-user at ci.uchicago.edu > > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Thu Jun 18 14:20:50 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 18 Jun 2015 12:20:50 -0700 Subject: [Swift-user] import position in swift script In-Reply-To: References: <1434649673.11316.0.camel@echo3> <1434650828.11812.4.camel@echo3> Message-ID: <1434655250.12474.25.camel@echo3> Hi, I don't think imports were meant to be used as a templating scheme, which is why they are restricted to be only at the start of a script. The alternative amounts to a dynamically scoped, no interface function call. So I don't think avoiding argument list bloat is a good enough reason to allow that, especially when you can package complex argument lists in a structure. Are the defs just values or do they also define behavior/functions? Mihael On Thu, 2015-06-18 at 18:18 +0000, Max Hutchinson wrote: > Not scattered; at the end. I will have ~10 use cases, each written as a > different defs file (i.e. defs_a.swift, defs_b.swift, etc). For each > use-case, the run.swift, which is a big foreach loop, is the same. Given > this structure, I can think of at least 4 organizations: > 1) run.swift imports "defs_a" or "defs_b". To change use cases, the > import in run.swift is edited. This is what I currently do but forces me > to either serialize inter-script execution or edit scripts that are in the > process of being executed. > 2) defs_a.swift, defs_b.swift, etc import "run". This seems clean and is > what I want to do. > 3) Wrapper scripts are created to contain both imports (i.e. a_wrap.swift > : ```import "defs_a"; import "run"```). This currently works, but results > in script bloat. > 4) Wrap the contents of run.swift in a procedure, import it at the top of > defs_{a,b,...}.swift, but then call it at the bottom. This is surely the > "right" way to do it, but its more effort and could have argument list > bloat. > Since each of these option is semantically identical, why not allow the > user to pick their preference? > > On Thu, Jun 18, 2015 at 1:07 PM Mihael Hategan wrote: > > > Yes, but I don't think it should. > > > > Why is it important to have import statements scattered through the > > script? > > > > Mihael > > > > On Thu, 2015-06-18 at 17:58 +0000, Max Hutchinson wrote: > > > Could that constraint be relaxed? > > > > > > On Thu, Jun 18, 2015 at 12:47 PM Mihael Hategan > > wrote: > > > > > > > Hi, > > > > > > > > Yes. Imports must come before other types of statements. > > > > > > > > Mihael > > > > > > > > On Thu, 2015-06-18 at 17:20 +0000, Max Hutchinson wrote: > > > > > Do all import statements need to come before executable statements? > > > > > > > > > > I have two scripts: defs.swift and run.swift. If run.swift starts > > with > > > > > import "defs"; > > > > > then everything works fine. If instead defs.swift ends with > > > > > import "run"; > > > > > I get this error: > > > > > Swift trunk git-rev: 1d512cbda070d363e8ec3cf16e77dfcddc6e66cb > > > > heads/master > > > > > 6377 > > > > > RunID: run011 > > > > > Could not compile SwiftScript source: line 24:12: unexpected token: ; > > > > > where line 24 is: > > > > > int foo = 1; > > > > > > > > > > Thanks, > > > > > Max > > > > > _______________________________________________ > > > > > Swift-user mailing list > > > > > Swift-user at ci.uchicago.edu > > > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > > > > > > > > > > > > > > > > > From maxhutch at gmail.com Thu Jun 18 14:48:04 2015 From: maxhutch at gmail.com (Max Hutchinson) Date: Thu, 18 Jun 2015 19:48:04 +0000 Subject: [Swift-user] import position in swift script In-Reply-To: <1434655250.12474.25.camel@echo3> References: <1434649673.11316.0.camel@echo3> <1434650828.11812.4.camel@echo3> <1434655250.12474.25.camel@echo3> Message-ID: I agree that it is a no interface function call. Maybe this is also helpful with not needing to explicitly collect outputs to get staged out? My understanding is that If run.swift contains a bunch of nested loops, one would have declare a bunch of [auto] outputs and remember to append everything to them (or they could be considered tmps and not staged out). There's value to being forced to be thoughtful, but it might also be nice to start with the trailing import while prototyping/debugging a script and then switch to a wrapper after the dust has settled. I'm focusing on direct staging, so the wrapper was straightforward. For reference: https://github.com/maxhutch/nek-swift/blob/alcf/debug.swift https://github.com/maxhutch/nek-swift/blob/alcf/nek.swift If imports are only going to be supported in the preamble, could that be added to the docs? http://swift-lang.org/guides/trunk/userguide/userguide.html#_imports On Thu, Jun 18, 2015 at 2:20 PM Mihael Hategan wrote: > Hi, > > I don't think imports were meant to be used as a templating scheme, > which is why they are restricted to be only at the start of a script. > The alternative amounts to a dynamically scoped, no interface function > call. So I don't think avoiding argument list bloat is a good enough > reason to allow that, especially when you can package complex argument > lists in a structure. > > Are the defs just values or do they also define behavior/functions? > > Mihael > > On Thu, 2015-06-18 at 18:18 +0000, Max Hutchinson wrote: > > Not scattered; at the end. I will have ~10 use cases, each written as a > > different defs file (i.e. defs_a.swift, defs_b.swift, etc). For each > > use-case, the run.swift, which is a big foreach loop, is the same. Given > > this structure, I can think of at least 4 organizations: > > 1) run.swift imports "defs_a" or "defs_b". To change use cases, the > > import in run.swift is edited. This is what I currently do but forces me > > to either serialize inter-script execution or edit scripts that are in > the > > process of being executed. > > 2) defs_a.swift, defs_b.swift, etc import "run". This seems clean and > is > > what I want to do. > > 3) Wrapper scripts are created to contain both imports (i.e. > a_wrap.swift > > : ```import "defs_a"; import "run"```). This currently works, but > results > > in script bloat. > > 4) Wrap the contents of run.swift in a procedure, import it at the top > of > > defs_{a,b,...}.swift, but then call it at the bottom. This is surely the > > "right" way to do it, but its more effort and could have argument list > > bloat. > > Since each of these option is semantically identical, why not allow the > > user to pick their preference? > > > > On Thu, Jun 18, 2015 at 1:07 PM Mihael Hategan > wrote: > > > > > Yes, but I don't think it should. > > > > > > Why is it important to have import statements scattered through the > > > script? > > > > > > Mihael > > > > > > On Thu, 2015-06-18 at 17:58 +0000, Max Hutchinson wrote: > > > > Could that constraint be relaxed? > > > > > > > > On Thu, Jun 18, 2015 at 12:47 PM Mihael Hategan > > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > Yes. Imports must come before other types of statements. > > > > > > > > > > Mihael > > > > > > > > > > On Thu, 2015-06-18 at 17:20 +0000, Max Hutchinson wrote: > > > > > > Do all import statements need to come before executable > statements? > > > > > > > > > > > > I have two scripts: defs.swift and run.swift. If run.swift > starts > > > with > > > > > > import "defs"; > > > > > > then everything works fine. If instead defs.swift ends with > > > > > > import "run"; > > > > > > I get this error: > > > > > > Swift trunk git-rev: 1d512cbda070d363e8ec3cf16e77dfcddc6e66cb > > > > > heads/master > > > > > > 6377 > > > > > > RunID: run011 > > > > > > Could not compile SwiftScript source: line 24:12: unexpected > token: ; > > > > > > where line 24 is: > > > > > > int foo = 1; > > > > > > > > > > > > Thanks, > > > > > > Max > > > > > > _______________________________________________ > > > > > > Swift-user mailing list > > > > > > Swift-user at ci.uchicago.edu > > > > > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > > > > > > > > > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Thu Jun 18 15:26:22 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 18 Jun 2015 13:26:22 -0700 Subject: [Swift-user] import position in swift script In-Reply-To: References: <1434649673.11316.0.camel@echo3> <1434650828.11812.4.camel@echo3> <1434655250.12474.25.camel@echo3> Message-ID: <1434659182.13760.25.camel@echo3> On Thu, 2015-06-18 at 19:48 +0000, Max Hutchinson wrote: > I agree that it is a no interface function call. Maybe this is also > helpful with not needing to explicitly collect outputs to get staged out? Good point. We, more or less, require that the app, perhaps through a wrapper script, fully define its interface through the command line. On the other hand, magical matching is allowed. If you have an output carefully mapped to what the application produces, it will work even without a wrapper, but it's impossible to make that work if two invocations produce two files with the same name. That should change. The reason it didn't is that we didn't have a good idea about how to make the collect specification nice in the case of non-trivial file lists. > My understanding is that If run.swift contains a bunch of nested loops, one > would have declare a bunch of [auto] outputs and remember to append > everything to them (or they could be considered tmps and not staged out). > There's value to being forced to be thoughtful, but it might also be nice > to start with the trailing import while prototyping/debugging a script and > then switch to a wrapper after the dust has settled. I'd rather add code to allow mapping in struct expressions, so that you could write something like this: params = { prefix: "debug", json: <"debug.json">, ... } test = sweep(params); and perhaps: with (params) { ... } > > I'm focusing on direct staging, so the wrapper was straightforward. For > reference: > https://github.com/maxhutch/nek-swift/blob/alcf/debug.swift > https://github.com/maxhutch/nek-swift/blob/alcf/nek.swift > > If imports are only going to be supported in the preamble, could that be > added to the docs? > http://swift-lang.org/guides/trunk/userguide/userguide.html#_imports We're in the process of revamping the docs. It will be in there. Mihael From maxhutch at gmail.com Thu Jun 18 18:54:04 2015 From: maxhutch at gmail.com (Max Hutchinson) Date: Thu, 18 Jun 2015 23:54:04 +0000 Subject: [Swift-user] import position in swift script In-Reply-To: <1434659182.13760.25.camel@echo3> References: <1434649673.11316.0.camel@echo3> <1434650828.11812.4.camel@echo3> <1434655250.12474.25.camel@echo3> <1434659182.13760.25.camel@echo3> Message-ID: That params struct sure looks a lot like a json dict. How about `params = readJSON("debug.json");`? On Thu, Jun 18, 2015 at 3:26 PM Mihael Hategan wrote: > On Thu, 2015-06-18 at 19:48 +0000, Max Hutchinson wrote: > > I agree that it is a no interface function call. Maybe this is also > > helpful with not needing to explicitly collect outputs to get staged out? > > Good point. We, more or less, require that the app, perhaps through a > wrapper script, fully define its interface through the command line. On > the other hand, magical matching is allowed. If you have an output > carefully mapped to what the application produces, it will work even > without a wrapper, but it's impossible to make that work if two > invocations produce two files with the same name. > > That should change. The reason it didn't is that we didn't have a good > idea about how to make the collect specification nice in the case of > non-trivial file lists. > > > My understanding is that If run.swift contains a bunch of nested loops, > one > > would have declare a bunch of [auto] outputs and remember to append > > everything to them (or they could be considered tmps and not staged out). > > There's value to being forced to be thoughtful, but it might also be nice > > to start with the trailing import while prototyping/debugging a script > and > > then switch to a wrapper after the dust has settled. > > I'd rather add code to allow mapping in struct expressions, so that you > could write something like this: > > params = { > prefix: "debug", > json: <"debug.json">, > ... > } > > test = sweep(params); > > and perhaps: > > with (params) { > ... > } > > > > > I'm focusing on direct staging, so the wrapper was straightforward. For > > reference: > > https://github.com/maxhutch/nek-swift/blob/alcf/debug.swift > > https://github.com/maxhutch/nek-swift/blob/alcf/nek.swift > > > > If imports are only going to be supported in the preamble, could that be > > added to the docs? > > http://swift-lang.org/guides/trunk/userguide/userguide.html#_imports > > We're in the process of revamping the docs. It will be in there. > > Mihael > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Thu Jun 18 19:01:01 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 18 Jun 2015 17:01:01 -0700 Subject: [Swift-user] import position in swift script In-Reply-To: References: <1434649673.11316.0.camel@echo3> <1434650828.11812.4.camel@echo3> <1434655250.12474.25.camel@echo3> <1434659182.13760.25.camel@echo3> Message-ID: <1434672061.16458.2.camel@echo3> On Thu, 2015-06-18 at 23:54 +0000, Max Hutchinson wrote: > That params struct sure looks a lot like a json dict. How about `params = > readJSON("debug.json");`? Not very far from our plans. If you scroll to the I/O section in https://github.com/swift-lang/swift-k/wiki/StandardLibrary, there's a mention of that. Mihael > > On Thu, Jun 18, 2015 at 3:26 PM Mihael Hategan wrote: > > > On Thu, 2015-06-18 at 19:48 +0000, Max Hutchinson wrote: > > > I agree that it is a no interface function call. Maybe this is also > > > helpful with not needing to explicitly collect outputs to get staged out? > > > > Good point. We, more or less, require that the app, perhaps through a > > wrapper script, fully define its interface through the command line. On > > the other hand, magical matching is allowed. If you have an output > > carefully mapped to what the application produces, it will work even > > without a wrapper, but it's impossible to make that work if two > > invocations produce two files with the same name. > > > > That should change. The reason it didn't is that we didn't have a good > > idea about how to make the collect specification nice in the case of > > non-trivial file lists. > > > > > My understanding is that If run.swift contains a bunch of nested loops, > > one > > > would have declare a bunch of [auto] outputs and remember to append > > > everything to them (or they could be considered tmps and not staged out). > > > There's value to being forced to be thoughtful, but it might also be nice > > > to start with the trailing import while prototyping/debugging a script > > and > > > then switch to a wrapper after the dust has settled. > > > > I'd rather add code to allow mapping in struct expressions, so that you > > could write something like this: > > > > params = { > > prefix: "debug", > > json: <"debug.json">, > > ... > > } > > > > test = sweep(params); > > > > and perhaps: > > > > with (params) { > > ... > > } > > > > > > > > I'm focusing on direct staging, so the wrapper was straightforward. For > > > reference: > > > https://github.com/maxhutch/nek-swift/blob/alcf/debug.swift > > > https://github.com/maxhutch/nek-swift/blob/alcf/nek.swift > > > > > > If imports are only going to be supported in the preamble, could that be > > > added to the docs? > > > http://swift-lang.org/guides/trunk/userguide/userguide.html#_imports > > > > We're in the process of revamping the docs. It will be in there. > > > > Mihael > > > > > > From jmtcollege at live.com Thu Jun 25 17:32:23 2015 From: jmtcollege at live.com (Jacob Taylor) Date: Thu, 25 Jun 2015 18:32:23 -0400 Subject: [Swift-user] Swift/T+Coasters Usage Message-ID: I am trying to use Swift/T+Coasters which I realize is not fully integrated. I have been able to get the Coasters service running locally, but whenever I try to execute any script that tries to dispatch to the Coasters service, it gives me the following error messageExecutor COASTER has no assigned workers while executing"error "Executor $exec_name has no assigned workers"" (procedure "turbine::check_can_execute" line 6) invoked from within"turbine::check_can_execute COASTER" (file "./swift-t-h.b4l.tic" line 127)and I have no idea what the problem is. I can send my log files, my config files, and the script itself if it will help. The documentation for using the two together is not very clear, do I need to manually launch the Coasters workers? Or should the service automatically launch them for me?Also, to clarify, for properly linking Coasters and Swift/T in the exm-settings.sh file before building, should the COASTER_INSTALL environment variable point to the installation directory or the source directory? --Jacob Taylor-- -------------- next part -------------- An HTML attachment was scrubbed... URL: From wozniak at mcs.anl.gov Thu Jun 25 16:53:45 2015 From: wozniak at mcs.anl.gov (Justin M Wozniak) Date: Thu, 25 Jun 2015 16:53:45 -0500 Subject: [Swift-user] Swift/T+Coasters Usage In-Reply-To: References: Message-ID: <558C7869.6030407@mcs.anl.gov> Ah- you need to set environment variable TURBINE_COASTER_WORKERS=1. I will improve the doc and the error message. On 6/25/2015 5:32 PM, Jacob Taylor wrote: > I am trying to use Swift/T+Coasters which I realize is not fully > integrated. I have been able to get the Coasters service running > locally, but whenever I try to execute any script that tries to > dispatch to the Coasters service, it gives me the following error message > > Executor COASTER has no assigned workers > while executing > "error "Executor $exec_name has no assigned workers"" > (procedure "turbine::check_can_execute" line 6) > invoked from within > "turbine::check_can_execute COASTER" > (file "./swift-t-h.b4l.tic" line 127) > > and I have no idea what the problem is. I can send my log files, my > config files, and the script itself if it will help. > > The documentation for using the two together is not very clear, do I > need to manually launch the Coasters workers? Or should the service > automatically launch them for me? > Also, to clarify, for properly linking Coasters and Swift/T in the > exm-settings.sh file before building, should the COASTER_INSTALL > environment variable point to the installation directory or the source > directory? > > --Jacob Taylor-- > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user -- Justin M Wozniak -------------- next part -------------- An HTML attachment was scrubbed... URL: