From yadudoc1729 at gmail.com Sat Sep 1 15:38:28 2012 From: yadudoc1729 at gmail.com (Yadu Nand) Date: Sun, 2 Sep 2012 02:08:28 +0530 Subject: [Swift-devel] MapReduce and swift Message-ID: Hi, I have a couple of questions about how mapreduce would look like if it were implemented in Swift. It would be great to hear ideas on these : 1) Google's MapReduce as well as Hadoop's makes certain assumptions about the data which are generated/used at each stage of processing . Mostly key-value pairs are used and the keys play a role in the reduce stage. Does this make sense in swift, or does the functional programming style make more sense here ? Swift mostly remains agnostic on file contents, so does it matter what is the format of the content of the files ? 2) Currently Swift would stage-in the results from a map job, and then stage-out to the reducers, while this could be shorted by just moving the results directly from the map nodes to the reduce nodes. What if we had semantics which would retrieve file locations rather than actual files and then choose to de-reference only when we needed to? (I do not think we have a method to do this already) say, type file_ptrs[]; // array of file pointers file_ptrs = map(foo, array); // here, we just get an array of file pointers result = reduce(bar, file_ptrs); // reduction would involve getting the actual files Maybe the above could be wrapped up as : result = mapreduce(foo, bar, array); -- Thanks and Regards, Yadu Nand B -------------- next part -------------- An HTML attachment was scrubbed... URL: From iraicu at cs.iit.edu Fri Sep 7 15:48:05 2012 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Fri, 07 Sep 2012 15:48:05 -0500 Subject: [Swift-devel] Extended Deadline: 5th Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS) 2012 -- at IEEE/ACM Supercomputing 2012 Message-ID: <504A5D85.1000909@cs.iit.edu> Call for Papers --------------------------------------------------------------------------------------- The 5th Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS) 2012 http://datasys.cs.iit.edu/events/MTAGS12/ --------------------------------------------------------------------------------------- November 12th, 2012 Salt Lake City, Utah, USA Co-located with with IEEE/ACM International Conference for High Performance Computing, Networking, Storage and Analysis (SC12) ======================================================================================= The 5th workshop on Many-Task Computing on Grids and Supercomputers (MTAGS) will provide the scientific community a dedicated forum for presenting new research, development, and deployment efforts of large-scale many-task computing (MTC) applications on large scale clusters, Grids, Supercomputers, and Cloud Computing infrastructure. MTC, the theme of the workshop encompasses loosely coupled applications, which are generally composed of many tasks (both independent and dependent tasks) to achieve some larger application goal. This workshop will cover challenges that can hamper efficiency and utilization in running applications on large-scale systems, such as local resource manager scalability and granularity, efficient utilization of raw hardware, parallel file system contention and scalability, data management, I/O management, reliability at scale, and application scalability. We welcome paper submissions on all theoretical, simulations, and systems topics related to MTC, but we give special consideration to papers addressing petascale to exascale challenges. Papers will be peer-reviewed, and accepted papers will be published in the workshop proceedings as part of the IEEE digital library (pending approval). The workshop will be co-located with the IEEE/ACM Supercomputing 2012 Conference in Salt Lake City Utah on November 12th, 2012. For more information, please see http://datasys.cs.iit.edu/events/MTAGS12/. For more information on past workshops, please see MTAGS11, MTAGS10, MTAGS09, and MTAGS08. We also ran a Special Issue on Many-Task Computing in the IEEE Transactions on Parallel and Distributed Systems (TPDS) which has appeared in June 2011; the proceedings can be found online at http://www.computer.org/portal/web/csdl/abs/trans/td/2011/06/ttd201106toc.htm. We, the workshop organizers, also published a highly relevant paper that defines Many-Task Computing which was published in MTAGS08, titled "Many-Task Computing for Grids and Supercomputers" (http://www.cs.iit.edu/~iraicu/research/publications/2008_MTAGS08_MTC.pdf); we encourage potential authors to read this paper. Topics --------------------------------------------------------------------------------------- We invite the submission of original work that is related to the topics below. The papers can be either short (4 pages) position papers, or long (8 pages) research papers. Topics of interest include (in the context of Many-Task Computing): * Compute Resource Management * Scheduling * Job execution frameworks * Local resource manager extensions * Performance evaluation of resource managers in use on large scale systems * Dynamic resource provisioning * Techniques to manage many-core resources and/or GPUs * Challenges and opportunities in running many-task workloads on HPC systems * Challenges and opportunities in running many-task workloads on Cloud Computing infrastructure * Storage architectures and implementations * Distributed file systems * Parallel file systems * Distributed meta-data management * Content distribution systems for large data * Data caching frameworks and techniques * Data management within and across data centers * Data-aware scheduling * Data-intensive computing applications * Eventual-consistency storage usage and management * Programming models and tools * Map-reduce and its generalizations * Many-task computing middleware and applications * Parallel programming frameworks * Ensemble MPI techniques and frameworks * Service-oriented science applications * Large-Scale Workflow Systems * Workflow system performance and scalability analysis * Scalability of workflow systems * Workflow infrastructure and e-Science middleware * Programming Paradigms and Models * Large-Scale Many-Task Applications * High-throughput computing (HTC) applications * Data-intensive applications * Quasi-supercomputing applications, deployments, and experiences * Performance Evaluation * Performance evaluation * Real systems * Simulations * Reliability of large systems * How MTC Addresses Challenges of Petascale and Exascale Computing * Concurency & Programmability * I/O & Memory * Energy * Resilience * Heterogeneity Paper Submission and Publication --------------------------------------------------------------------------------------- Authors are invited to submit papers with unpublished, original work of not more than 8 pages of double column text using single spaced 10 point size on 8.5 x 11 inch pages, as per IEEE 8.5 x 11 manuscript guidelines; document templates can be found at http://www.ieee.org/conferences_events/conferences/publishing/templates.html. We are also seeking position papers of no more than 4 pages in length. The final 4/8 page papers in PDF format must be submitted online at https://cmt.research.microsoft.com/MTAGS2012/ before the deadline. Papers will be peer-reviewed, and accepted papers will be published in the workshop proceedings as part of the IEEE digital library (pending approval). Notifications of the paper decisions will be sent out by October 12th, 2012. Selected excellent work may be eligible for additional post-conference publication as journal articles or book chapters, such as the previous Special Issue on Many-Task Computing in the IEEE Transactions on Parallel and Distributed Systems (TPDS) which has appeared in June 2011. Submission implies the willingness of at least one of the authors to register and present the paper. For more information, please http://datasys.cs.iit.edu/events/MTAGS12/, or send email to mtags12-chairs at datasys.cs.iit.edu. Important Dates --------------------------------------------------------------------------------------- * Abstract submission: September 10th, 2012 (11:59PM PST) * Paper submission: September 24th, 2012 (11:59PM PST) * Acceptance notification: October 12th, 2012 * Final papers due: November 7th, 2012 Committee Members --------------------------------------------------------------------------------------- Workshop Chairs (mtags12-chairs at datasys.cs.iit.edu) * Ioan Raicu, Illinois Institute of Technology & Argonne National Laboratory * Ian Foster, University of Chicago & Argonne National Laboratory * Yong Zhao, University of Electronic Science and Technology of China Steering Committee * David Abramson, Monash University, Australia * Jack Dongara, University of Tennessee, USA * Geoffrey Fox, Indiana University, USA * Manish Parashar, Rutgers University, USA * Marc Snir, Argonne National Laboratory & University of Illinois at Urbana Champaign, USA * Xian-He Sun, Illinois Institute of Technology, USA * Weimin Zheng, Tsinghua University, China Publicity Chair (mtags12-publicity at datasys.cs.iit.edu) * Zhao Zhang, University of Chicago, USA Program Committee Chair (mtags12-pc-chair at datasys.cs.iit.edu) * Justin Wozniak, Argonne National Laboratory, USA Technical Committee * Roger Barga, Microsoft Research, USA * Mihai Budiu, Microsoft Research, USA * Kyle Chard, University of Chicago, USA * Yong Chen, Texas Tech University, USA * Evangelinos Constantinos, Massachusetts Institute of Technology, USA * John Dennis, National Center for Atmospheric Research, USA * Catalin Dumitrescu, Fermi National Labs, USA * Dennis Gannon, Microsoft Research, USA * Indranil Gupta, University of Illinois at Urbana Champaign, USA * Florin Isaila, Universidad Carlos III de Madrid, Spain * Kamil Iskra, Argonne National Laboratory, USA * Alexandru Iosup, Delft University of Technology, Netherlands * Hui Jin, Oracle Corporation, USA * Daniel S. Katz, University of Chicago & Argonne National Laboratory, USA * Carl Kesselman, University of Southern California, USA * Zhiling Lan, Illinois Institute of Technology, USA * Mike Lang, Los Alamos National Laboratory, USA * Gregor von Laszewski, Indiana University, USA * Reagan Moore, University of North Carolina, Chappel Hill, USA * Jose Moreira, IBM Research, USA * Chris Moretti, Princeton University, USA * David O'Hallaron, Carnegie Mellon University, Intel Labs, USA * Marlon Pierce, Indiana University, USA * Judy Qiu, Indiana University, USA * Lavanya Ramakrishnan, Lawrence Berkeley National Laboratory, USA * Kui Ren, SUNY Buffalo, USA * Matei Ripeanu, University of British Columbia, Canada * Karen Schuchardt, Pacific Northwest National Laboratory, USA * Wei Tang, Argonne National Laboratory, USA * Valerie Taylor, Texas A&M, USA * Douglas Thain University of Notre Dame, USA * Edward Walker, Whitworth University, USA * Matthew Woitaszek, Occipital, Inc., USA * Ken Yocum, University of California, San Diego, USA * Zhifeng Yun, Louisiana State University, USA * Zhao Zhang, University of Chicago, USA * Ziming Zheng, Illinois Institute of Technology, USA -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ ================================================================= ================================================================= From iraicu at cs.iit.edu Fri Sep 7 17:13:20 2012 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Fri, 07 Sep 2012 17:13:20 -0500 Subject: [Swift-devel] eScience 2012 Program now available Message-ID: <504A7180.9080600@cs.iit.edu> The Conference Program for the 8th IEEE International Conference on eScience (eScience 2012) is now available at http://www.ci.uchicago.edu/escience2012/. All eScience conference and workshops will be held at the Hyatt Regency Chicago. We have reserved a block of rooms at the hotel with a room rate of $205 per night + tax. Please make your reservations before September 16 at https://resweb.passkey.com/go/ESCIENCE2012 or by phone at +1-800-233-1234 from the US or +1-312-565-1234 from outside the US. Reference "e-Science" when making reservations by phone. The hotel is completely sold out on Sunday night (10/7) due to major events being held in Chicago over the weekend. We have secured a block of rooms at the Hyatt Regency O'Hare for $190 + tax and will provide transportation to the conference hotel on Monday morning. You will be able to make your reservation at the Hyatt O'Hare upon registration for the conference or via email to kristih at ci.uchicago.edu . Advance registration is available until September 28 - see http://www.ci.uchicago.edu/escience2012/registration.php. ACKNOWLEDGEMENTS-- We gratefully acknowledge the support of the following organizations-- Gold Level: Microsoft Research Silver Level: CSIRO Australia Bronze Level: Argonne National Laboratory EMC Indiana University Media: HPCwire Also: The Computation Institute Cray Conference Sponsors: IEEE The IEEE Computer Society -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From iraicu at cs.iit.edu Fri Sep 7 19:11:07 2012 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Fri, 07 Sep 2012 19:11:07 -0500 Subject: [Swift-devel] Call for Participation: The 9th ICAC 2012 Conference - September 17-21 in San Jose CA Message-ID: <504A8D1B.5000800@cs.iit.edu> ********************************************************************** CALL FOR PARTICIPATION ====================== The 9th ACM International Conference on Autonomic Computing (ICAC 2012) San Jose, California, USA September 17-21, 2012 http://icac2012.cs.fiu.edu Sponsored by ACM ********************************************************************** Online registration is open at http://icac2012.cs.fiu.edu/registration.shtm Reduced fees are available for those registering by September 9, 2012 (extended). This year's technical program features: - 3 distinguished keynote speakers: * Dr. Amin Vahdat, Google/UCSD * Dr. Subutai Ahmad (VP Engineering), Numenta * Dr. Eitan Frachtenberg, Facebook - 24 outstanding technical papers (15 full + 9 short): * covering core and emerging topics such as clouds, virtualization, control, monitoring and diagnosis, and energy * Half of the papers involve authors from industry or government labs - 4 co-located workshops covering hot topics in: * Feedback Computing * Self-Aware Internet of Things * Management of Big Data Systems * Federated Clouds The Conference will be held at the Fairmont Hotel in downtown San Jose, CA, USA. ********************************************************************** IMPORTANT DATES =============== Early registration deadline: September 9, 2012 (extended) Hotel special rate deadline: September 7, 2012 ********************************************************************** CORPORATE SPONSORS ================== Gold Level Partner: IBM Conference partners: VMware, HP, Neustar PhD Student Sponsor: Google Other Sponsor: NEC Labs, Microsoft ********************************************************************** PROGRAM ======= ====================================================================== MONDAY, SEPTEMBER 17, 2012 - WORKSHOPS Feedback Computing 2012 Self-Aware Internet of Things 2012 ====================================================================== TUESDAY, SEPTEMBER 18, 2012 - MAIN CONFERENCE 8:00AM ? 8:45AM Registration 8:45AM ? 9:00AM Welcome Remarks 9:00AM ? 10:00AM Keynote Talk I Symbiosis in Scale Out Networking and Data Management Amin Vahdat, Google and University of California, San Diego 10:00AM ? 10.30AM Break 10:30AM ? 12:00PM Session : Virtualization Net-Cohort: Detecting and Managing VM Ensembles in Virtualized Data Centers Liting Hu, Karsten Schwan (Georgia Institute of Technology); Ajay Gulati (VMware); Junjie Zhang, Chengwei Wang (Georgia Institute of Technology) Application-aware Cross-layer Virtual Machine Resource Management Lixi Wang, Jing Xu, Ming Zhao (Florida International University) Shifting GEARS to Enable Guest-context Virtual Services Kyle Hale, Lei Xia, Peter Dinda (Northwestern University) 12:00PM-1:30PM Lunch ---------------------------------------------------------------------- 1:30PM - 3:30PM Session: Performance and Resource Management When Average is Not Average: Large Response Time Fluctuations in N-tier Systems Qingyang Wang (Georgia Institute of Technology); Yasuhiko Kanemasa, Motoyuki Kawaba (Fujitsu Laboratories Ltd.); Calton Pu (Georgia Institute of Technology) Provisioning Multi-tier Cloud Applications Using Statistical Bounds on Sojourn Time Upendra Sharma, Prashant Shenoy, Don Towsley (University of Massachusetts Amherst) Automated Profiling and Resource Management of Pig Programs for Meeting Service Level Objectives Zhuoyao Zhang (University of Pennsylvania); Ludmila Cherkasova (Hewlett-Packard Labs); Abhishek Verma (University of Illinois at Urbana-Champaign); Boon Thau Loo (University of Pennsylvania) AROMA: Automated Resource Allocation and Configuration of MapReduce Environment in the Cloud Palden Lama, Xiaobo Zhou (University of Colorado at Colorado Springs) 3:30PM-4:00PM Break 4:00PM ? 5:15PM Short Papers I Locomotion at Location: When the Rubber hits the Road Gerold Hoelzl, Marc Kurz, Alois Ferscha (Johannes Kepler University Linz, Austria) An Autonomic Resource Provisioning Framework for Mobile Computing Grids Hariharasudhan Viswanathan, Eun Kyung Lee, Ivan Rodero, Dario Pompili (Rutgers University) A Self-Tuning Self-Optimizing Approach for Automated Network Anomaly Detection Systems Dennis Ippoliti, Xiaobo Zhou (University of Colorado at Colorado Springs) Offline and On-Demand Event Correlation for Operations Management of Large Scale IT Systems Chetan Gupta (Hewlett-Packard Labs) PowerTracer: Tracing Requests in Multi-tier Services to Diagnose Energy Inefficiency Gang Lu, Jianfeng Zhan (Institute of Computing Technology, Chinese Academy of Sciences); Haining Wang (College of William and Mary); Lin Yuan (Institute of Computing Technology, Chinese Academy of Sciences); Chuliang Weng (Shanghai Jiao Tong University, China) 6:00PM - 9:00PM Conference Dinner ====================================================================== WEDNESDAY, SEPTEMBER 19 - MAIN CONFERENCE 8:00AM ? 9:00AM Registration 9:00AM ? 10:00AM Keynote Talk II Automated Machine Learning For Autonomic Computing Subutai Ahmad, VP Engineering, Numenta 10:00AM - 10:30AM Break 10:30AM ? 12:00PM Session: Control-Based Approaches Budget-based Control for Interactive Services with Adaptive Execution Yuxiong He, Zihao Ye, Qiang Fu, Sameh Elnikety (Microsoft Research) On the Design of Decentralized Control Architectures for Workload Consolidation in Large-Scale Server Clusters Rui Wang, Nagarajan Kandasamy (Drexel University) Transactional Auto Scaler: Elastic Scaling of In-Memory Transactional Data Grids Diego Didona, Paolo Romano (Instituto Superior T?cnico/INESC-ID); Sebastiano Peluso, Francesco Quaglia (Sapienza, Universit? di Roma) 12:00PM - 1:30PM Lunch ---------------------------------------------------------------------- 1:30PM - 2:30PM Session: Energy Adaptive Green Hosting Nan Deng, Christopher Stewart, Jaimie Kelley (The Ohio State University); Daniel Gmach, Martin Arlitt (Hewlett Packard Labs) Dynamic Energy-Aware Capacity Provisioning for Cloud Computing Environments Qi Zhang, Mohamed Faten Zhani (University of Waterloo); Shuo Zhang (National University of Defense Technology); Quanyan Zhu (University of Illinois at Urbana-Champaign); Raouf Boutaba (University of Waterloo); Joseph L. Hellerstein (Google, Inc.) 2:30PM - 3:30PM Short Papers II VESPA: Multi-Layered Self-Protection for Cloud Resources Aur?lien Wailly, Marc Lacoste (Orange Labs); Herv? Debar (T?l?com SudParis) Usage Patterns in Multi-tenant Data Centers: a Temporal Perspective Robert Birke, Lydia Y. Chen (IBM Research Zurich Lab); Evgenia Smirni (College of William and Mary) Toward Fast Eventual Consistency with Performance Guarantees Feng Yan (College of William and Mary); Alma Riska (EMC Corporation); Evgenia Smirni (College of William and Mary) Optimal Autoscaling in the IaaS Cloud Hamoun Ghanbari, Bradley Simmons, Marin Litoiu, Cornel Barna (York University); Gabriel Iszlai (IBM Toronto) 3:30PM ? 4:00PM Break 4:00PM - 6:00PM Poster and Demo Session 6:00PM-9:00PM Conference Outing (tentative) ====================================================================== THURSDAY, SEPTEMBER 20, 2012 - MAIN CONFERENCE 8:00AM ? 9:00AM Registration 9:00AM ? 10:00AM Keynote Talk III High Efficiency at Web Scale Eitan Frachtenberg, Facebook 10:00AM-10:30AM Break 10:30AM - 12:00PM Session: Diagnosis and Monitoring Chair: TBD 3-Dimensional Root Cause Diagnosis via Co-analysi Ziming Zheng, Li Yu, Zhiling Lan (Illinois Institute of Technology); Terry Jones (Oak Ridge National Laboratory) UBL: Unsupervised Behavior Learning for Predicting Performance Anomalies in Virtualized Cloud Systems Daniel J. Dean, Hiep Nguyen, Xiaohui Gu (North Carolina State University) Evaluating Compressive Sampling Strategies for Performance Monitoring of Data Centers Tingshan Huang, Nagarajan Kandasamy, Harish Sethu (Drexel University) 12:00PM Adjourn ====================================================================== FRIDAY, SEPTEMBER 21, 2012 - WORKSHOPS Management of Big Data Systems 2012 Federated Clouds 2012 ********************************************************************** ORGANIZERS ========== GENERAL CHAIR Dejan Milojicic, HP Labs PROGRAM CHAIRS Dongyan Xu, Purdue University Vanish Talwar, HP Labs INDUSTRY CHAIR Xiaoyun Zhu, VMware WORKSHOPS CHAIR Fred Douglis, EMC POSTERS/DEMO/EXHIBITS CHAIR Eno Thereska, Microsoft Research FINANCE CHAIR Michael Kozuch, Intel LOCAL ARRANGEMENT CHAIR Jessica Blaine PUBLICITY CHAIRS Daniel Batista, University of S?o Paulo Vartan Padaryan, ISP/Russian Academy of Sci. Ioan Raicu, Illinois Inst. of Technology Jianfeng Zhan, ICT/Chinese Academy of Sci. Ming Zhao, Florida Intl. University PROGRAM COMMITTEE Tarek Abdelzaher, UIUC Umesh Bellur, IIT, Bombay Ken Birman, Cornell University Rajkumar Buyya, Univ. of Melbourne Rocky Chang, Hong Kong Polytechnic University Yuan Chen, HP Labs Alva Couch, Tufts University Peter Dinda, Northwestern University Fred Douglis, EMC Renato Figueiredo, University of Florida Mohamed Hefeeda, QCRI Joe Hellerstein, Google Geoff Jiang, NEC Labs Jeff Kephart, IBM Research Emre Kiciman, Microsoft Research Fabio Kon, University of S?o Paulo Mike Kozuch, Intel Labs Dejan Milojicic, HP Labs Klara Nahrstedt, UIUC Priya Narasimhan, CMU Manish Parashar, Rutgers University Ioan Raicu, Illinois Inst. of Technology Omer Rana, Cardiff University Masoud Sadjadi, Florida Intl. University Richard Schlichting, AT&T Labs Hartmut Schmeck, KIT Karsten Schwan, Georgia Tech Onn Shehory, IBM Research Eno Thereska, Microsoft Research Xiaoyun Zhu, VMware ********************************************************************** -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ ================================================================= ================================================================= From wilde at mcs.anl.gov Fri Sep 14 09:54:34 2012 From: wilde at mcs.anl.gov (Michael Wilde) Date: Fri, 14 Sep 2012 09:54:34 -0500 (CDT) Subject: [Swift-devel] Fwd: [Swift-user] Swift progress reports are too frequent In-Reply-To: Message-ID: <729621291.16483.1347634474241.JavaMail.root@zimbra.anl.gov> Im moving this thread to swift-devel. Mihael or David, can you tell us (or determine) why the progress message frequency changed from 30 secs to 3 secs? Should revert back to 30? Thanks, - Mike ----- Forwarded Message ----- From: "Jonathan Margoliash" To: swift-user at ci.uchicago.edu Sent: Friday, September 14, 2012 8:47:56 AM Subject: [Swift-user] Swift progress reports are too frequent Hello swift support, I just downloaded the most recent version of the swift trunk, and now Swift prints to a progress report every 3 seconds. (Something of the form "Progress: .... ") Beforehand, swift printed something every 30 seconds, and that made scrolling through swift's output much more manageable. How can I change this back, so that swift delivers reports much less often? Jonathan _______________________________________________ Swift-user mailing list Swift-user at ci.uchicago.edu https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory From iraicu at cs.iit.edu Fri Sep 14 16:33:06 2012 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Fri, 14 Sep 2012 16:33:06 -0500 Subject: [Swift-devel] HPDC 2013 Call for Workshops Message-ID: <5053A292.70300@cs.iit.edu> HPDC 2013 Call for Workshops http://www.hpdc.org/2013/workshops/call-for-workshops/ The organizers of the 22nd International ACM Symposium on High-Performance Parallel and Distributed Computing (HPDC'13) call for proposals for workshops to be held with HPDC'13. The workshops will be held on June 17-18, 2013. Workshops should provide forums for discussion among researchers and practitioners on focused topics or emerging research areas relevant to the HPDC community. Organizers may structure workshops as they see fit, including invited talks, panel discussions, presentations of work in progress, fully peer-reviewed papers, or some combination. Workshops could be scheduled for half a day or a full day, depending on interest, space constraints, and organizer preference. Organizers should design workshops for approximately 20-40 participants, to balance impact and effective discussion. Workshop proposals must be sent in PDF format to the HPDC'13 Workshops Chair, Abhishek Chandra (Email: chandra AT cs DOT umn DOT edu) with the subject line "HPDC 2013 Workshop Proposal", and should include: - The name and acronym of the workshop - A description (0.5-1 page) of the theme of the workshop - A description (one paragraph) of the relation between the theme of the workshop and of HPDC - A list of topics of interest - The names and affiliations of the workshop organizers, and if applicable, of a significant portion of the program committee - A description of the expected structure of the workshop (papers, invited talks, panel discussions, etc.) - Data about previous offerings of the workshop (if any), including the attendance, the numbers of papers or presentations submitted and accepted, and the links to the corresponding websites - A publicity plan for attracting submissions and attendees. Please also include expected number of submissions, accepted papers, and attendees that you anticipate for a successful workshop. Due to publication deadlines, workshops must operate within roughly the following timeline: papers due mid February (2-3 weeks after the HPDC deadline), and selected and sent to the publisher by mid April. Important dates: Workshop Proposals Due: October 25, 2012 Notifications: November 2, 2012 Workshop CFPs Online and Distributed: November 25, 2012 -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From iraicu at cs.iit.edu Fri Sep 14 16:58:32 2012 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Fri, 14 Sep 2012 16:58:32 -0500 Subject: [Swift-devel] CFP: 13th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) 2013 Message-ID: <5053A888.6010001@cs.iit.edu> **** CALL FOR PAPERS **** The 13th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid 2013) Delft University of Technology, Delft, the Netherlands May 13-16, 2013 http://www.pds.ewi.tudelft.nl/ccgrid2013 Rapid advances in architectures, networks, and systems and middleware technologies are leading to new concepts in and platforms for computing, ranging from Clusters and Grids to Clouds and Datacenters. CCGrid is a series of very successful conferences, sponsored by the IEEE Computer Society Technical Committee on Scalable Computing (TCSC) and the ACM, with the overarching goal of bringing together international researchers, developers, and users to provide an international forum to present leading research activities and results on a broad range of topics related to these concepts and platforms, and their applications. The conference features keynotes, technical presentations, workshops, tutorials, and posters, as well as the SCALE challenge featuring live demonstrations. In 2013, CCGrid will come to the Netherlands for the first time, and will be held in Delft, a historical, picturesque city that is less than one hour away from Amsterdam-Schiphol airport. The main conference will be held on May 14-16 (Tuesday to Thursday), with tutorials and affiliated workshops taking place on May 13 (Monday). **** IMPORTANT DATES **** Papers Due: 12 November 2012 Author Notifications: 24 January 2013 Final Papers Due: 22 February 2013 **** TOPICS OF INTEREST **** CCGrid 2013 will have a focus on important and immediate issues that are significantly influencing all aspects of cluster, cloud and grid computing. Topics of interest include, but are not limited to: * Applications and Experiences: Applications to real and complex problems in science, engineering, business, and society; User studies; Experiences with large-scale deployments, systems, or applications * Architecture: System architectures, design and deployment; Power and cooling; Security and reliability; High availability solutions * Autonomic Computing and Cyberinfrastructure: Self-managed behavior, models and technologies; Autonomic paradigms and systems (control-based, bio-inspired, emergent, etc.); Bio-inspired optimizations and computing * Cloud Computing: Cloud architectures; Software tools and techniques for clouds * Multicore and Accelerator-based Computing: Software and application techniques to utilize multicore architectures and accelerators in clusters, grids, and clouds * Performance Modeling and Evaluation: Performance prediction and modeling; Monitoring and evaluation tools; Analysis of system and application performance; Benchmarks and testbeds * Programming Models, Systems, and Fault-Tolerant Computing: Programming models and environments for cluster, cloud, and grid computing; Fault-tolerant systems, programs and algorithms; Systems software to support efficient computing * Scheduling and Resource Management: Techniques to schedule jobs and resources on cluster, cloud, and grid computing platforms; SLA definition and enforcement **** PAPER SUBMISSION GUIDELINES **** Authors are invited to submit papers electronically in PDF format. Submitted manuscripts should be structured as technical papers and may not exceed 8 letter-size (8.5 x 11) pages including figures, tables and references using the IEEE format for conference proceedings. Submissions not conforming to these guidelines may be returned without review. Authors should make sure that their file will print on a printer that uses letter-size (8.5 x 11) paper. The official language of the conference is English. All manuscripts will be reviewed and will be judged on correctness, originality, technical strength, significance, quality of presentation, and interest and relevance to the conference attendees. Submitted papers must represent original unpublished research that is not currently under review for any other conference or journal. Papers not following these guidelines will be rejected without review and further action may be taken, including (but not limited to) notifications sent to the heads of the institutions of the authors and sponsors of the conference. Submissions received after the due date, exceeding the page limit, or not appropriately structured may not be considered. Authors may contact the conference chairs for more information. The proceedings will be published through the IEEE Computer Society Press, USA, and will be made available online through the IEEE Digital Library. **** CALL FOR TUTORIAL AND WORKSHOP PROPOSALS **** Tutorials and workshops affiliated with CCGrid 2013 will be held on May 13 (Monday). For more information on the tutorials and workshops and for the complete Call for Tutorial and Workshop Proposals, please see the conference website. **** GENERAL CHAIR **** Dick Epema, Delft University of Technology, the Netherlands **** PROGRAM CHAIR **** Thomas Fahringer, University of Innsbruck, Austria **** PROGRAM VICE-CHAIRS **** Rosa Badia, Barcelona Supercomputing Center, Spain Henri Bal, Vrije Universiteit, the Netherlands Marios Dikaiakos, University of Cyprus, Cyprus Kirk Cameron, VirginiaTech, USA Daniel Katz, University of Chicago & Argonne Nat Lab, USA Kate Keahey, Argonne National Laboratory, USA Martin Schulz, Lawrence Livermore National Laboratory, USA Douglas Thain, University of Notre Dame, USA Cheng-Zhong Xu, Shenzhen Inst. of Advanced Techn, China **** WORKSHOPS CO-CHAIRS **** Shantenu Jha, Rutgers and Louisana State University, USA Ioan Raicu, Illinois Institute of Technology, USA **** TOTORIALS CHAIR **** Radu Prodan, University of Innsbruck, Austria **** DOCTORAL SYMPOSIUM CO-CHAIRS **** Yogesh Simmhan, University of Southern California, USA Ana Varbanescu, Delft Univ of Technology, the Netherlands **** SUBMISSIONS AND PROCEEDINGS CHAIR **** Pavan Balaji, Argonne National Laboratory, USA **** FINANCE AND REGISTRATION CHAIR **** Alexandru Iosup, Delft Univ of Technology, the Netherlands **** PUBLICITY CHAIRS **** Nazareno Andrade, University Federal de Campina Grance, Brazil Gabriel Antoniu, INRIA, France Bahman Javadi, University of Western Sysney, Australia Ioan Raicu, Illinois Institute of Technology and Argonne National Laboratory, USA Kin Choong Yow, Shenzhen Inst. of Advanced Technology, China **** CYBER CHAIR **** Stephen van der Laan, Delft University of Technology, the Netherlands -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From iraicu at cs.iit.edu Fri Sep 14 18:59:42 2012 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Fri, 14 Sep 2012 18:59:42 -0500 Subject: [Swift-devel] Call for Workshops: IEEE/ACM CCGrid 2013 Message-ID: <5053C4EE.6070700@cs.iit.edu> **** CALL FOR WORKSHOPS **** The 13th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid 2013) Delft University of Technology, Delft, the Netherlands May 13-16, 2013 http://www.pds.ewi.tudelft.nl/ccgrid2013 http://www.pds.ewi.tudelft.nl/ccgrid2013/calls/workshops/ Workshop proposals are invited for CCGrid 2013 on specific aspects of Grid, Cloud and Cluster Computing, particularly relating to the subject areas indicated by the topics below. We encourage workshops that will discuss fundamental research issues driven by academic interests or more applied industrial or commercial concerns. The format of the workshop will be determined by the organizers. Workshops can vary in length from a half day to a full day. Having more than one co-organizer for a workshop is strongly advised. Workshop proceedings will be published as part of the CCGrid 2013 proceedings. So, it is very important that high quality workshops are accepted, and that workshop chairs observe strict quality standards, no more than 50% acceptance, in the selection of papers for their events. Workshop attendees are required to register for the main conference. IMPORTANT DATES AND PROPOSAL SUBMISSION: --------------------------------------------------------------------------- Workshop proposals and any enquiries should be sent by e-mail to the workshop chairs. Proposals should be submitted in PDF format (printable on A4 paper). Workshop Proposals Due: October 20, 2012 Notification: November 3, 2012 Workshops: May 13-16 2013 PROPOSAL REQUIREMENTS: ------------------------------------------- Proposals for workshops should be between 2 to 5 pages in length. They should contain the following information: o Title and brief technical description of the workshop, specifying the goals and the technical issues that will be its focus. o A brief abstract of the workshop (less than 200 words), intended for the CCGrid 2013 web site. o A brief description of why and to whom the workshop is of interest. o A list of related workshops or similar events held in the last 3 years or to be held in 2012/2013. o The names and contact information (web page, email address) of the proposed organizing committee. This committee should consist of two or three people knowledgeable about the technical issues to be addressed, preferably not members of the same institution. A brief description of the qualifications of the proposed organizing committee with respect to organizing this workshop (e.g., papers published in the proposed topic area, previous workshop organization, other relevant information). o Link to a preliminary web site of the workshop and a preliminary call for papers o A list of committed and proposed PC members. RESPONSIBILITIES: ------------------------------ Each workshop organizing committee will be responsible for the following: o Producing a web page and a "Call for Papers/Participation" for their workshop. The URL should be sent to the workshop co-chairs for CCGrid 2013. o The call must make it clear that the workshop is open to all members of the Distributed Computing community. It should mention that at least one author of each accepted submission must attend the workshop and that all workshop participants must pay the conference fee. o Finally, it should also clearly describe the process by which the Organizing Committee will select the participants. o Ensure that all workshop papers are a maximum of 6 pages in length (in IEEE format). It is the responsibility of the workshop organizers to ensure that this page limit has been adhered to. Additional pages may be purchased (in some circumstances) subject to approval of the proceedings chair. o Provide a brief description of the workshop for the conference web page and program. o Selecting the participants and the format of the workshop. The publication of proceedings will be by the IEEE in the same volume as the main conference. Camera-ready due date for the accepted workshop papers will be the same as the main conference. Therefore, workshop organizers should set the acceptance notification date at least 2 weeks earlier than the camera-ready due date. All other details can be up to workshop organizers to set, such as advertising the workshop beyond the conference web page and assistance in producing a camera-ready version of the workshop proceedings. Important Note: ----------------------- Workshop organizers must ensure that suitable quality measures have been taken to ensure that the accepted papers are of high quality. All papers must be reviewed by an International Programme Committee (with a minimum of 3 reviews per paper). The CCGrid 2013 Organizing Committee will be responsible for the following: o Providing a link to a workshop's local page. o Providing logistics support and a meeting place for the workshop. o In conjunction with the organizers, determining the workshop date and time. o Providing copies of the workshop proceedings to attendees. Important Note: ----------------------- The CCGrid 2013 Organizing Committee may decide, if the workshop is too small (i.e. does not attract enough submissions) to merge it with another workshop. So we encourage workshop organizers to attract a large community. In extreme situations we may also cancel workshops if there are not enough submissions. Conference Topics of Interest and Area Keywords: ------------------------------------------------------------------------- Topics of interest to the conference include (but are not restricted to): o Autonomic Grid Computing o Content Distribution Networks o Cloud Computing o Cluster Computing o Grid Computing o Peer-2-Peer Computing o Multi-Core Systems o Grid and Cloud Testbeds o Semantic Grids o Web 2.0 Technologies o Workflow Tools and Applications o Programming Models o Energy Management in Data Centers o Resource Management o Service Level Agreements and Scheduling o Tools and Environments o Scientific Instruments and Grid Computing o Application areas: HealthCare/Life Sciences, Engineering, etc. WORKSHOP CHAIRS: -------------------------------- Please send your proposals to both workshop chairs. o Ioan Raicu (iraicu at cs.iit.edu), Illinois Institute of Technology, USA o Shantenu Jha (shantenu.jha at rutgers.edu), Rutgers University, USA -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ ================================================================= ================================================================= From wilde at mcs.anl.gov Tue Sep 18 12:44:03 2012 From: wilde at mcs.anl.gov (Michael Wilde) Date: Tue, 18 Sep 2012 12:44:03 -0500 (CDT) Subject: [Swift-devel] Eradicating the "Failed to transfer wrapper log error" In-Reply-To: Message-ID: <331467765.23177.1347990243868.JavaMail.root@zimbra.anl.gov> I think its high time to turn this message into a clear diagnosis for the user of the root cause of the problem: Swift tried to run an app an the attempt failed due to: - permission - executable not found - etc Ive filed this as bug 837 and assigned to David for now. We can discuss the approach and who all needs to get involved, via that ticket. - Mike ----- Forwarded Message ----- From: "Lorenzo Pesce" To: swift-support at ci.uchicago.edu Sent: Tuesday, September 18, 2012 11:11:20 AM Subject: [swift-support] Failed to transfer wrapper log error Hi -- I have no idea what this means. I am working at directory: /lustre/beagle/GCNet/RG/Oreo/o080513_B using configuration script: /lustre/beagle/GCNet/bin/Swift/swift_demo_real_test.sh This is the error I see on my screen. I could not figure out what the problem might be from the log files. Progress: time: Tue, 18 Sep 2012 16:07:30 +0000 Initializing:14879 Selecting site:1 Progress: time: Tue, 18 Sep 2012 16:07:33 +0000 Selecting site:14879 Initializing site shared directory:1 Progress: time: Tue, 18 Sep 2012 16:07:34 +0000 Selecting site:14856 Stage in:3 Submitting:20 Submitted:1 Progress: time: Tue, 18 Sep 2012 16:07:45 +0000 Selecting site:14856 Submitted:24 Progress: time: Tue, 18 Sep 2012 16:07:57 +0000 Selecting site:14856 Submitted:23 Active:1 Failed to transfer wrapper log for job DemoRealWrapper-6xi07ayk Failed to transfer wrapper log for job DemoRealWrapper-qvi07ayk Failed to transfer wrapper log for job DemoRealWrapper-xvi07ayk Failed to transfer wrapper log for job DemoRealWrapper-4xi07ayk Failed to transfer wrapper log for job DemoRealWrapper-svi07ayk Failed to transfer wrapper log for job DemoRealWrapper-2xi07ayk Failed to transfer wrapper log for job DemoRealWrapper-vvi07ayk Failed to transfer wrapper log for job DemoRealWrapper-mvi07ayk Failed to transfer wrapper log for job DemoRealWrapper-pvi07ayk Failed to transfer wrapper log for job DemoRealWrapper-lvi07ayk Failed to transfer wrapper log for job DemoRealWrapper-tvi07ayk Failed to transfer wrapper log for job DemoRealWrapper-5xi07ayk Failed to transfer wrapper log for job DemoRealWrapper-7xi07ayk Failed to transfer wrapper log for job DemoRealWrapper-jvi07ayk Failed to transfer wrapper log for job DemoRealWrapper-1xi07ayk Failed to transfer wrapper log for job DemoRealWrapper-yvi07ayk Failed to transfer wrapper log for job DemoRealWrapper-ovi07ayk Failed to transfer wrapper log for job DemoRealWrapper-0xi07ayk Failed to transfer wrapper log for job DemoRealWrapper-3xi07ayk Failed to transfer wrapper log for job DemoRealWrapper-rvi07ayk Failed to transfer wrapper log for job DemoRealWrapper-uvi07ayk Failed to transfer wrapper log for job DemoRealWrapper-kvi07ayk Failed to transfer wrapper log for job DemoRealWrapper-nvi07ayk Failed to transfer wrapper log for job DemoRealWrapper-zvi07ayk Thanks a lot for your help, Lorenzo _______________________________________________ swift-support mailing list swift-support at ci.uchicago.edu https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-support -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory From davidk at ci.uchicago.edu Tue Sep 18 13:33:27 2012 From: davidk at ci.uchicago.edu (David Kelly) Date: Tue, 18 Sep 2012 13:33:27 -0500 (CDT) Subject: [Swift-devel] Dynamic profiles and count In-Reply-To: <1649123873.57360.1347992179094.JavaMail.root@zimbra-mb2.anl.gov> Message-ID: <1966915315.57505.1347993207886.JavaMail.root@zimbra-mb2.anl.gov> Hello, I have been working on a namd scaling test using Swift. I am using the plain PBS provider at the moment (no coasters). The swift script sets a minimum number of nodes, a maximum number of nodes, iterates through those values, then uses dynamic profiles to change the value of 'count' to modify the number of nodes to request. Here is the script: --- type file; app (file out, file err) namd_wrapper (int numnodes, file psf_file, file pdb_file, file coord_restart_file, file velocity_restart_file, file system_restart_file) { profile "count" = numnodes; namd_wrapper @psf_file @pdb_file @coord_restart_file @velocity_restart_file @system_restart_file stdout=@out stderr=@err; } # Range of nodes to test on int minNodes=1; int maxNodes=2; int delta=1; # Files file psf <"h0_solvion.psf">; file pdb <"h0_solvion.pdb">; file coord_restart <"h0_eq.0.restart.coor">; file velocity_restart <"h0_eq.0.restart.vel">; file system_restart <"h0_eq.0.restart.xsc">; foreach nodes in [minNodes:maxNodes:delta] { file output ; file error ; (output, error) = namd_wrapper(nodes, psf, pdb, coord_restart, velocity_restart, system_restart); } --- In sites.xml, I also set the jobtype to "single" so it doesn't start a worker on each node (namd uses MPI). The problem that I'm running into is that, as is, dynamic profiles do not seem to allow you to modify the value for count. I have a workaround for this which involves removing references to "count" in GridExec.java, Execute.java, and TCProfile.java. This works for me in terms of this script, and it works with a few other simple catsn type scripts I've tested. I just wanted to double check to make sure this wouldn't cause any other issues before committing. Here are the changes: Index: modules/karajan/src/org/globus/cog/karajan/workflow/nodes/grid/GridExec.java =================================================================== --- modules/karajan/src/org/globus/cog/karajan/workflow/nodes/grid/GridExec.java (revision 3472) +++ modules/karajan/src/org/globus/cog/karajan/workflow/nodes/grid/GridExec.java (working copy) @@ -56,7 +56,7 @@ public static final Arg A_STDIN = new Arg.Optional("stdin"); public static final Arg A_PROVIDER = new Arg.Optional("provider"); public static final Arg A_SECURITY_CONTEXT = new Arg.Optional("securitycontext"); - public static final Arg A_COUNT = new Arg.Optional("count"); + // public static final Arg A_COUNT = new Arg.Optional("count"); public static final Arg A_HOST_COUNT = new Arg.Optional("hostcount"); public static final Arg A_JOBTYPE = new Arg.Optional("jobtype"); public static final Arg A_MAXTIME = new Arg.Optional("maxtime"); @@ -86,7 +86,8 @@ static { setArguments(GridExec.class, new Arg[] { A_EXECUTABLE, A_ARGS, A_ARGUMENTS, A_HOST, A_STDOUT, A_STDERR, A_STDOUTLOCATION, A_STDERRLOCATION, A_STDIN, A_PROVIDER, - A_COUNT, A_HOST_COUNT, A_JOBTYPE, A_MAXTIME, A_MAXWALLTIME, A_MAXCPUTIME, + // A_COUNT, + A_HOST_COUNT, A_JOBTYPE, A_MAXTIME, A_MAXWALLTIME, A_MAXCPUTIME, A_ENVIRONMENT, A_QUEUE, A_PROJECT, A_MINMEMORY, A_MAXMEMORY, A_REDIRECT, A_SECURITY_CONTEXT, A_DIRECTORY, A_NATIVESPEC, A_DELEGATION, A_ATTRIBUTES, C_ENVIRONMENT, A_FAIL_ON_JOB_ERROR, A_BATCH, C_STAGEIN, C_STAGEOUT, C_CLEANUP, @@ -346,7 +347,7 @@ } } - protected final static Arg[] MISC_ATTRS = new Arg[] { A_COUNT, A_HOST_COUNT, A_JOBTYPE, + protected final static Arg[] MISC_ATTRS = new Arg[] { A_HOST_COUNT, A_JOBTYPE, A_MAXTIME, A_MAXWALLTIME, A_MAXCPUTIME, A_QUEUE, A_PROJECT, A_MINMEMORY, A_MAXMEMORY }; protected void setMiscAttributes(JobSpecification js, VariableStack stack) Index: src/org/griphyn/vdl/karajan/lib/TCProfile.java =================================================================== --- src/org/griphyn/vdl/karajan/lib/TCProfile.java (revision 5930) +++ src/org/griphyn/vdl/karajan/lib/TCProfile.java (working copy) @@ -63,7 +63,6 @@ static { PROFILE_T = new HashMap(); - PROFILE_T.put("count", GridExec.A_COUNT); PROFILE_T.put("jobtype", GridExec.A_JOBTYPE); PROFILE_T.put("maxcputime", GridExec.A_MAXCPUTIME); PROFILE_T.put("maxmemory", GridExec.A_MAXMEMORY); Index: src/org/griphyn/vdl/karajan/lib/Execute.java =================================================================== --- src/org/griphyn/vdl/karajan/lib/Execute.java (revision 5930) +++ src/org/griphyn/vdl/karajan/lib/Execute.java (working copy) @@ -47,7 +47,7 @@ static { setArguments(Execute.class, new Arg[] { A_EXECUTABLE, A_ARGS, A_ARGUMENTS, A_HOST, A_STDOUT, A_STDERR, A_STDOUTLOCATION, A_STDERRLOCATION, A_STDIN, A_PROVIDER, - A_COUNT, A_HOST_COUNT, A_JOBTYPE, A_MAXTIME, A_MAXWALLTIME, A_MAXCPUTIME, + A_HOST_COUNT, A_JOBTYPE, A_MAXTIME, A_MAXWALLTIME, A_MAXCPUTIME, A_ENVIRONMENT, A_QUEUE, A_PROJECT, A_MINMEMORY, A_MAXMEMORY, A_REDIRECT, A_SECURITY_CONTEXT, A_DIRECTORY, A_NATIVESPEC, A_DELEGATION, A_ATTRIBUTES, C_ENVIRONMENT, A_FAIL_ON_JOB_ERROR, A_BATCH, A_REPLICATION_GROUP, From hategan at mcs.anl.gov Tue Sep 18 16:58:12 2012 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 18 Sep 2012 14:58:12 -0700 Subject: [Swift-devel] Dynamic profiles and count In-Reply-To: <1966915315.57505.1347993207886.JavaMail.root@zimbra-mb2.anl.gov> References: <1966915315.57505.1347993207886.JavaMail.root@zimbra-mb2.anl.gov> Message-ID: <1348005492.17528.4.camel@blabla> I'm not sure. The way I understand things, your change modifies the way execute is invoked from: execute(executable, ..., count=n) to: execute(executable, ..., attributes=map(map:entry("count", n), ...)) But the semantics are the same. The task object, in both cases, will have an attribute named "count" equal to n. Can you send me the full diff of your changes? Mihael On Tue, 2012-09-18 at 13:33 -0500, David Kelly wrote: > Hello, > > I have been working on a namd scaling test using Swift. I am using the plain PBS provider at the moment (no coasters). The swift script sets a minimum number of nodes, a maximum number of nodes, iterates through those values, then uses dynamic profiles to change the value of 'count' to modify the number of nodes to request. Here is the script: > > --- > type file; > > app (file out, file err) namd_wrapper (int numnodes, file psf_file, file pdb_file, file coord_restart_file, > file velocity_restart_file, file system_restart_file) > { > profile "count" = numnodes; > namd_wrapper @psf_file @pdb_file @coord_restart_file @velocity_restart_file @system_restart_file stdout=@out stderr=@err; > } > > # Range of nodes to test on > int minNodes=1; > int maxNodes=2; > int delta=1; > > # Files > file psf <"h0_solvion.psf">; > file pdb <"h0_solvion.pdb">; > file coord_restart <"h0_eq.0.restart.coor">; > file velocity_restart <"h0_eq.0.restart.vel">; > file system_restart <"h0_eq.0.restart.xsc">; > > foreach nodes in [minNodes:maxNodes:delta] { > file output ; > file error ; > (output, error) = namd_wrapper(nodes, psf, pdb, coord_restart, velocity_restart, system_restart); > } > --- > > In sites.xml, I also set the jobtype to "single" so it doesn't start a worker on each node (namd uses MPI). > > The problem that I'm running into is that, as is, dynamic profiles do not seem to allow you to modify the value for count. I have a workaround for this which involves removing references to "count" in GridExec.java, Execute.java, and TCProfile.java. This works for me in terms of this script, and it works with a few other simple catsn type scripts I've tested. I just wanted to double check to make sure this wouldn't cause any other issues before committing. Here are the changes: > > Index: modules/karajan/src/org/globus/cog/karajan/workflow/nodes/grid/GridExec.java > =================================================================== > --- modules/karajan/src/org/globus/cog/karajan/workflow/nodes/grid/GridExec.java (revision 3472) > +++ modules/karajan/src/org/globus/cog/karajan/workflow/nodes/grid/GridExec.java (working copy) > @@ -56,7 +56,7 @@ > public static final Arg A_STDIN = new Arg.Optional("stdin"); > public static final Arg A_PROVIDER = new Arg.Optional("provider"); > public static final Arg A_SECURITY_CONTEXT = new Arg.Optional("securitycontext"); > - public static final Arg A_COUNT = new Arg.Optional("count"); > + // public static final Arg A_COUNT = new Arg.Optional("count"); > public static final Arg A_HOST_COUNT = new Arg.Optional("hostcount"); > public static final Arg A_JOBTYPE = new Arg.Optional("jobtype"); > public static final Arg A_MAXTIME = new Arg.Optional("maxtime"); > @@ -86,7 +86,8 @@ > static { > setArguments(GridExec.class, new Arg[] { A_EXECUTABLE, A_ARGS, A_ARGUMENTS, A_HOST, > A_STDOUT, A_STDERR, A_STDOUTLOCATION, A_STDERRLOCATION, A_STDIN, A_PROVIDER, > - A_COUNT, A_HOST_COUNT, A_JOBTYPE, A_MAXTIME, A_MAXWALLTIME, A_MAXCPUTIME, > + // A_COUNT, > + A_HOST_COUNT, A_JOBTYPE, A_MAXTIME, A_MAXWALLTIME, A_MAXCPUTIME, > A_ENVIRONMENT, A_QUEUE, A_PROJECT, A_MINMEMORY, A_MAXMEMORY, A_REDIRECT, > A_SECURITY_CONTEXT, A_DIRECTORY, A_NATIVESPEC, A_DELEGATION, A_ATTRIBUTES, > C_ENVIRONMENT, A_FAIL_ON_JOB_ERROR, A_BATCH, C_STAGEIN, C_STAGEOUT, C_CLEANUP, > @@ -346,7 +347,7 @@ > } > } > > - protected final static Arg[] MISC_ATTRS = new Arg[] { A_COUNT, A_HOST_COUNT, A_JOBTYPE, > + protected final static Arg[] MISC_ATTRS = new Arg[] { A_HOST_COUNT, A_JOBTYPE, > A_MAXTIME, A_MAXWALLTIME, A_MAXCPUTIME, A_QUEUE, A_PROJECT, A_MINMEMORY, A_MAXMEMORY }; > > protected void setMiscAttributes(JobSpecification js, VariableStack stack) > > Index: src/org/griphyn/vdl/karajan/lib/TCProfile.java > =================================================================== > --- src/org/griphyn/vdl/karajan/lib/TCProfile.java (revision 5930) > +++ src/org/griphyn/vdl/karajan/lib/TCProfile.java (working copy) > @@ -63,7 +63,6 @@ > > static { > PROFILE_T = new HashMap(); > - PROFILE_T.put("count", GridExec.A_COUNT); > PROFILE_T.put("jobtype", GridExec.A_JOBTYPE); > PROFILE_T.put("maxcputime", GridExec.A_MAXCPUTIME); > PROFILE_T.put("maxmemory", GridExec.A_MAXMEMORY); > > Index: src/org/griphyn/vdl/karajan/lib/Execute.java > =================================================================== > --- src/org/griphyn/vdl/karajan/lib/Execute.java (revision 5930) > +++ src/org/griphyn/vdl/karajan/lib/Execute.java (working copy) > @@ -47,7 +47,7 @@ > static { > setArguments(Execute.class, new Arg[] { A_EXECUTABLE, A_ARGS, A_ARGUMENTS, A_HOST, > A_STDOUT, A_STDERR, A_STDOUTLOCATION, A_STDERRLOCATION, A_STDIN, A_PROVIDER, > - A_COUNT, A_HOST_COUNT, A_JOBTYPE, A_MAXTIME, A_MAXWALLTIME, A_MAXCPUTIME, > + A_HOST_COUNT, A_JOBTYPE, A_MAXTIME, A_MAXWALLTIME, A_MAXCPUTIME, > A_ENVIRONMENT, A_QUEUE, A_PROJECT, A_MINMEMORY, A_MAXMEMORY, A_REDIRECT, > A_SECURITY_CONTEXT, A_DIRECTORY, A_NATIVESPEC, A_DELEGATION, A_ATTRIBUTES, > C_ENVIRONMENT, A_FAIL_ON_JOB_ERROR, A_BATCH, A_REPLICATION_GROUP, > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel From wozniak at mcs.anl.gov Tue Sep 18 21:40:54 2012 From: wozniak at mcs.anl.gov (Justin M Wozniak) Date: Tue, 18 Sep 2012 21:40:54 -0500 Subject: [Swift-devel] Dynamic profiles and count In-Reply-To: <1348005492.17528.4.camel@blabla> References: <1966915315.57505.1347993207886.JavaMail.root@zimbra-mb2.anl.gov> <1348005492.17528.4.camel@blabla> Message-ID: <505930B6.5090401@mcs.anl.gov> Just to clarify- this is based on changes already in svn regarding dynamic profiles. That's why the map is there. http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html#_dynamic_profiles David is trying to solve a specific issue with using dynamic profiles and "count": we need to know all about "count" to solve this. This is a paste of my recent email to David about dynamic profiles and issues with "count": Ok, the place to look is in the Swift repo: svn diff -r5206:5207 Apparently, no changes were made to CoG. The dynamic profiles should be visible in the KML. From there, you really need to know your Karajan semantics. The attributes are known to execute2() and thus execute() and Swift's Execute. Now Java. You should be able to see the attributes in GridExec. They are applied to the Task's JobSpecification. These good places to add some trace-level logging. Then, the Task is sent to PBSExecutor. You could add logging here too to make sure the attributes made it. One thing to consider is the possibility that the attribute is being overwritten by some other component. If you add logging to JobSpecification.setAttribute(), you might be able to find that. On 9/18/2012 4:58 PM, Mihael Hategan wrote: > I'm not sure. The way I understand things, your change modifies the way > execute is invoked from: > > execute(executable, ..., count=n) > > to: > > execute(executable, ..., attributes=map(map:entry("count", n), ...)) > > But the semantics are the same. The task object, in both cases, will > have an attribute named "count" equal to n. > > Can you send me the full diff of your changes? > > Mihael > > On Tue, 2012-09-18 at 13:33 -0500, David Kelly wrote: >> Hello, >> >> I have been working on a namd scaling test using Swift. I am using the plain PBS provider at the moment (no coasters). The swift script sets a minimum number of nodes, a maximum number of nodes, iterates through those values, then uses dynamic profiles to change the value of 'count' to modify the number of nodes to request. Here is the script: >> >> --- >> type file; >> >> app (file out, file err) namd_wrapper (int numnodes, file psf_file, file pdb_file, file coord_restart_file, >> file velocity_restart_file, file system_restart_file) >> { >> profile "count" = numnodes; >> namd_wrapper @psf_file @pdb_file @coord_restart_file @velocity_restart_file @system_restart_file stdout=@out stderr=@err; >> } >> >> # Range of nodes to test on >> int minNodes=1; >> int maxNodes=2; >> int delta=1; >> >> # Files >> file psf <"h0_solvion.psf">; >> file pdb <"h0_solvion.pdb">; >> file coord_restart <"h0_eq.0.restart.coor">; >> file velocity_restart <"h0_eq.0.restart.vel">; >> file system_restart <"h0_eq.0.restart.xsc">; >> >> foreach nodes in [minNodes:maxNodes:delta] { >> file output ; >> file error ; >> (output, error) = namd_wrapper(nodes, psf, pdb, coord_restart, velocity_restart, system_restart); >> } >> --- >> >> In sites.xml, I also set the jobtype to "single" so it doesn't start a worker on each node (namd uses MPI). >> >> The problem that I'm running into is that, as is, dynamic profiles do not seem to allow you to modify the value for count. I have a workaround for this which involves removing references to "count" in GridExec.java, Execute.java, and TCProfile.java. This works for me in terms of this script, and it works with a few other simple catsn type scripts I've tested. I just wanted to double check to make sure this wouldn't cause any other issues before committing. Here are the changes: >> >> Index: modules/karajan/src/org/globus/cog/karajan/workflow/nodes/grid/GridExec.java >> =================================================================== >> --- modules/karajan/src/org/globus/cog/karajan/workflow/nodes/grid/GridExec.java (revision 3472) >> +++ modules/karajan/src/org/globus/cog/karajan/workflow/nodes/grid/GridExec.java (working copy) >> @@ -56,7 +56,7 @@ >> public static final Arg A_STDIN = new Arg.Optional("stdin"); >> public static final Arg A_PROVIDER = new Arg.Optional("provider"); >> public static final Arg A_SECURITY_CONTEXT = new Arg.Optional("securitycontext"); >> - public static final Arg A_COUNT = new Arg.Optional("count"); >> + // public static final Arg A_COUNT = new Arg.Optional("count"); >> public static final Arg A_HOST_COUNT = new Arg.Optional("hostcount"); >> public static final Arg A_JOBTYPE = new Arg.Optional("jobtype"); >> public static final Arg A_MAXTIME = new Arg.Optional("maxtime"); >> @@ -86,7 +86,8 @@ >> static { >> setArguments(GridExec.class, new Arg[] { A_EXECUTABLE, A_ARGS, A_ARGUMENTS, A_HOST, >> A_STDOUT, A_STDERR, A_STDOUTLOCATION, A_STDERRLOCATION, A_STDIN, A_PROVIDER, >> - A_COUNT, A_HOST_COUNT, A_JOBTYPE, A_MAXTIME, A_MAXWALLTIME, A_MAXCPUTIME, >> + // A_COUNT, >> + A_HOST_COUNT, A_JOBTYPE, A_MAXTIME, A_MAXWALLTIME, A_MAXCPUTIME, >> A_ENVIRONMENT, A_QUEUE, A_PROJECT, A_MINMEMORY, A_MAXMEMORY, A_REDIRECT, >> A_SECURITY_CONTEXT, A_DIRECTORY, A_NATIVESPEC, A_DELEGATION, A_ATTRIBUTES, >> C_ENVIRONMENT, A_FAIL_ON_JOB_ERROR, A_BATCH, C_STAGEIN, C_STAGEOUT, C_CLEANUP, >> @@ -346,7 +347,7 @@ >> } >> } >> >> - protected final static Arg[] MISC_ATTRS = new Arg[] { A_COUNT, A_HOST_COUNT, A_JOBTYPE, >> + protected final static Arg[] MISC_ATTRS = new Arg[] { A_HOST_COUNT, A_JOBTYPE, >> A_MAXTIME, A_MAXWALLTIME, A_MAXCPUTIME, A_QUEUE, A_PROJECT, A_MINMEMORY, A_MAXMEMORY }; >> >> protected void setMiscAttributes(JobSpecification js, VariableStack stack) >> >> Index: src/org/griphyn/vdl/karajan/lib/TCProfile.java >> =================================================================== >> --- src/org/griphyn/vdl/karajan/lib/TCProfile.java (revision 5930) >> +++ src/org/griphyn/vdl/karajan/lib/TCProfile.java (working copy) >> @@ -63,7 +63,6 @@ >> >> static { >> PROFILE_T = new HashMap(); >> - PROFILE_T.put("count", GridExec.A_COUNT); >> PROFILE_T.put("jobtype", GridExec.A_JOBTYPE); >> PROFILE_T.put("maxcputime", GridExec.A_MAXCPUTIME); >> PROFILE_T.put("maxmemory", GridExec.A_MAXMEMORY); >> >> Index: src/org/griphyn/vdl/karajan/lib/Execute.java >> =================================================================== >> --- src/org/griphyn/vdl/karajan/lib/Execute.java (revision 5930) >> +++ src/org/griphyn/vdl/karajan/lib/Execute.java (working copy) >> @@ -47,7 +47,7 @@ >> static { >> setArguments(Execute.class, new Arg[] { A_EXECUTABLE, A_ARGS, A_ARGUMENTS, A_HOST, >> A_STDOUT, A_STDERR, A_STDOUTLOCATION, A_STDERRLOCATION, A_STDIN, A_PROVIDER, >> - A_COUNT, A_HOST_COUNT, A_JOBTYPE, A_MAXTIME, A_MAXWALLTIME, A_MAXCPUTIME, >> + A_HOST_COUNT, A_JOBTYPE, A_MAXTIME, A_MAXWALLTIME, A_MAXCPUTIME, >> A_ENVIRONMENT, A_QUEUE, A_PROJECT, A_MINMEMORY, A_MAXMEMORY, A_REDIRECT, >> A_SECURITY_CONTEXT, A_DIRECTORY, A_NATIVESPEC, A_DELEGATION, A_ATTRIBUTES, >> C_ENVIRONMENT, A_FAIL_ON_JOB_ERROR, A_BATCH, A_REPLICATION_GROUP, >> _______________________________________________ >> Swift-devel mailing list >> Swift-devel at ci.uchicago.edu >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel -- Justin M Wozniak From hategan at mcs.anl.gov Tue Sep 18 22:42:51 2012 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 18 Sep 2012 20:42:51 -0700 Subject: [Swift-devel] Dynamic profiles and count In-Reply-To: <505930B6.5090401@mcs.anl.gov> References: <1966915315.57505.1347993207886.JavaMail.root@zimbra-mb2.anl.gov> <1348005492.17528.4.camel@blabla> <505930B6.5090401@mcs.anl.gov> Message-ID: <1348026171.20771.0.camel@blabla> Ah, thanks. I can check and see what's happening. On Tue, 2012-09-18 at 21:40 -0500, Justin M Wozniak wrote: > Just to clarify- this is based on changes already in svn regarding > dynamic profiles. That's why the map is there. > > http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html#_dynamic_profiles > > David is trying to solve a specific issue with using dynamic profiles > and "count": we need to know all about "count" to solve this. > > This is a paste of my recent email to David about dynamic profiles and > issues with "count": > > Ok, the place to look is in the Swift repo: svn diff -r5206:5207 > > Apparently, no changes were made to CoG. > > The dynamic profiles should be visible in the KML. > > From there, you really need to know your Karajan semantics. The > attributes are known to execute2() and thus execute() and Swift's Execute. > > Now Java. You should be able to see the attributes in GridExec. They > are applied to the Task's JobSpecification. These good places to add > some trace-level logging. > > Then, the Task is sent to PBSExecutor. You could add logging here too > to make sure the attributes made it. > > One thing to consider is the possibility that the attribute is being > overwritten by some other component. If you add logging to > JobSpecification.setAttribute(), you might be able to find that. > > On 9/18/2012 4:58 PM, Mihael Hategan wrote: > > I'm not sure. The way I understand things, your change modifies the way > > execute is invoked from: > > > > execute(executable, ..., count=n) > > > > to: > > > > execute(executable, ..., attributes=map(map:entry("count", n), ...)) > > > > But the semantics are the same. The task object, in both cases, will > > have an attribute named "count" equal to n. > > > > Can you send me the full diff of your changes? > > > > Mihael > > > > On Tue, 2012-09-18 at 13:33 -0500, David Kelly wrote: > >> Hello, > >> > >> I have been working on a namd scaling test using Swift. I am using the plain PBS provider at the moment (no coasters). The swift script sets a minimum number of nodes, a maximum number of nodes, iterates through those values, then uses dynamic profiles to change the value of 'count' to modify the number of nodes to request. Here is the script: > >> > >> --- > >> type file; > >> > >> app (file out, file err) namd_wrapper (int numnodes, file psf_file, file pdb_file, file coord_restart_file, > >> file velocity_restart_file, file system_restart_file) > >> { > >> profile "count" = numnodes; > >> namd_wrapper @psf_file @pdb_file @coord_restart_file @velocity_restart_file @system_restart_file stdout=@out stderr=@err; > >> } > >> > >> # Range of nodes to test on > >> int minNodes=1; > >> int maxNodes=2; > >> int delta=1; > >> > >> # Files > >> file psf <"h0_solvion.psf">; > >> file pdb <"h0_solvion.pdb">; > >> file coord_restart <"h0_eq.0.restart.coor">; > >> file velocity_restart <"h0_eq.0.restart.vel">; > >> file system_restart <"h0_eq.0.restart.xsc">; > >> > >> foreach nodes in [minNodes:maxNodes:delta] { > >> file output ; > >> file error ; > >> (output, error) = namd_wrapper(nodes, psf, pdb, coord_restart, velocity_restart, system_restart); > >> } > >> --- > >> > >> In sites.xml, I also set the jobtype to "single" so it doesn't start a worker on each node (namd uses MPI). > >> > >> The problem that I'm running into is that, as is, dynamic profiles do not seem to allow you to modify the value for count. I have a workaround for this which involves removing references to "count" in GridExec.java, Execute.java, and TCProfile.java. This works for me in terms of this script, and it works with a few other simple catsn type scripts I've tested. I just wanted to double check to make sure this wouldn't cause any other issues before committing. Here are the changes: > >> > >> Index: modules/karajan/src/org/globus/cog/karajan/workflow/nodes/grid/GridExec.java > >> =================================================================== > >> --- modules/karajan/src/org/globus/cog/karajan/workflow/nodes/grid/GridExec.java (revision 3472) > >> +++ modules/karajan/src/org/globus/cog/karajan/workflow/nodes/grid/GridExec.java (working copy) > >> @@ -56,7 +56,7 @@ > >> public static final Arg A_STDIN = new Arg.Optional("stdin"); > >> public static final Arg A_PROVIDER = new Arg.Optional("provider"); > >> public static final Arg A_SECURITY_CONTEXT = new Arg.Optional("securitycontext"); > >> - public static final Arg A_COUNT = new Arg.Optional("count"); > >> + // public static final Arg A_COUNT = new Arg.Optional("count"); > >> public static final Arg A_HOST_COUNT = new Arg.Optional("hostcount"); > >> public static final Arg A_JOBTYPE = new Arg.Optional("jobtype"); > >> public static final Arg A_MAXTIME = new Arg.Optional("maxtime"); > >> @@ -86,7 +86,8 @@ > >> static { > >> setArguments(GridExec.class, new Arg[] { A_EXECUTABLE, A_ARGS, A_ARGUMENTS, A_HOST, > >> A_STDOUT, A_STDERR, A_STDOUTLOCATION, A_STDERRLOCATION, A_STDIN, A_PROVIDER, > >> - A_COUNT, A_HOST_COUNT, A_JOBTYPE, A_MAXTIME, A_MAXWALLTIME, A_MAXCPUTIME, > >> + // A_COUNT, > >> + A_HOST_COUNT, A_JOBTYPE, A_MAXTIME, A_MAXWALLTIME, A_MAXCPUTIME, > >> A_ENVIRONMENT, A_QUEUE, A_PROJECT, A_MINMEMORY, A_MAXMEMORY, A_REDIRECT, > >> A_SECURITY_CONTEXT, A_DIRECTORY, A_NATIVESPEC, A_DELEGATION, A_ATTRIBUTES, > >> C_ENVIRONMENT, A_FAIL_ON_JOB_ERROR, A_BATCH, C_STAGEIN, C_STAGEOUT, C_CLEANUP, > >> @@ -346,7 +347,7 @@ > >> } > >> } > >> > >> - protected final static Arg[] MISC_ATTRS = new Arg[] { A_COUNT, A_HOST_COUNT, A_JOBTYPE, > >> + protected final static Arg[] MISC_ATTRS = new Arg[] { A_HOST_COUNT, A_JOBTYPE, > >> A_MAXTIME, A_MAXWALLTIME, A_MAXCPUTIME, A_QUEUE, A_PROJECT, A_MINMEMORY, A_MAXMEMORY }; > >> > >> protected void setMiscAttributes(JobSpecification js, VariableStack stack) > >> > >> Index: src/org/griphyn/vdl/karajan/lib/TCProfile.java > >> =================================================================== > >> --- src/org/griphyn/vdl/karajan/lib/TCProfile.java (revision 5930) > >> +++ src/org/griphyn/vdl/karajan/lib/TCProfile.java (working copy) > >> @@ -63,7 +63,6 @@ > >> > >> static { > >> PROFILE_T = new HashMap(); > >> - PROFILE_T.put("count", GridExec.A_COUNT); > >> PROFILE_T.put("jobtype", GridExec.A_JOBTYPE); > >> PROFILE_T.put("maxcputime", GridExec.A_MAXCPUTIME); > >> PROFILE_T.put("maxmemory", GridExec.A_MAXMEMORY); > >> > >> Index: src/org/griphyn/vdl/karajan/lib/Execute.java > >> =================================================================== > >> --- src/org/griphyn/vdl/karajan/lib/Execute.java (revision 5930) > >> +++ src/org/griphyn/vdl/karajan/lib/Execute.java (working copy) > >> @@ -47,7 +47,7 @@ > >> static { > >> setArguments(Execute.class, new Arg[] { A_EXECUTABLE, A_ARGS, A_ARGUMENTS, A_HOST, > >> A_STDOUT, A_STDERR, A_STDOUTLOCATION, A_STDERRLOCATION, A_STDIN, A_PROVIDER, > >> - A_COUNT, A_HOST_COUNT, A_JOBTYPE, A_MAXTIME, A_MAXWALLTIME, A_MAXCPUTIME, > >> + A_HOST_COUNT, A_JOBTYPE, A_MAXTIME, A_MAXWALLTIME, A_MAXCPUTIME, > >> A_ENVIRONMENT, A_QUEUE, A_PROJECT, A_MINMEMORY, A_MAXMEMORY, A_REDIRECT, > >> A_SECURITY_CONTEXT, A_DIRECTORY, A_NATIVESPEC, A_DELEGATION, A_ATTRIBUTES, > >> C_ENVIRONMENT, A_FAIL_ON_JOB_ERROR, A_BATCH, A_REPLICATION_GROUP, > >> _______________________________________________ > >> Swift-devel mailing list > >> Swift-devel at ci.uchicago.edu > >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > From hategan at mcs.anl.gov Tue Sep 18 23:25:33 2012 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 18 Sep 2012 21:25:33 -0700 Subject: [Swift-devel] Dynamic profiles and count In-Reply-To: <1348026171.20771.0.camel@blabla> References: <1966915315.57505.1347993207886.JavaMail.root@zimbra-mb2.anl.gov> <1348005492.17528.4.camel@blabla> <505930B6.5090401@mcs.anl.gov> <1348026171.20771.0.camel@blabla> Message-ID: <1348028733.21339.0.camel@blabla> David, Can you try r5934? Mihael On Tue, 2012-09-18 at 20:42 -0700, Mihael Hategan wrote: > Ah, thanks. I can check and see what's happening. > > On Tue, 2012-09-18 at 21:40 -0500, Justin M Wozniak wrote: > > Just to clarify- this is based on changes already in svn regarding > > dynamic profiles. That's why the map is there. > > > > http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html#_dynamic_profiles > > > > David is trying to solve a specific issue with using dynamic profiles > > and "count": we need to know all about "count" to solve this. > > > > This is a paste of my recent email to David about dynamic profiles and > > issues with "count": > > > > Ok, the place to look is in the Swift repo: svn diff -r5206:5207 > > > > Apparently, no changes were made to CoG. > > > > The dynamic profiles should be visible in the KML. > > > > From there, you really need to know your Karajan semantics. The > > attributes are known to execute2() and thus execute() and Swift's Execute. > > > > Now Java. You should be able to see the attributes in GridExec. They > > are applied to the Task's JobSpecification. These good places to add > > some trace-level logging. > > > > Then, the Task is sent to PBSExecutor. You could add logging here too > > to make sure the attributes made it. > > > > One thing to consider is the possibility that the attribute is being > > overwritten by some other component. If you add logging to > > JobSpecification.setAttribute(), you might be able to find that. > > > > On 9/18/2012 4:58 PM, Mihael Hategan wrote: > > > I'm not sure. The way I understand things, your change modifies the way > > > execute is invoked from: > > > > > > execute(executable, ..., count=n) > > > > > > to: > > > > > > execute(executable, ..., attributes=map(map:entry("count", n), ...)) > > > > > > But the semantics are the same. The task object, in both cases, will > > > have an attribute named "count" equal to n. > > > > > > Can you send me the full diff of your changes? > > > > > > Mihael > > > > > > On Tue, 2012-09-18 at 13:33 -0500, David Kelly wrote: > > >> Hello, > > >> > > >> I have been working on a namd scaling test using Swift. I am using the plain PBS provider at the moment (no coasters). The swift script sets a minimum number of nodes, a maximum number of nodes, iterates through those values, then uses dynamic profiles to change the value of 'count' to modify the number of nodes to request. Here is the script: > > >> > > >> --- > > >> type file; > > >> > > >> app (file out, file err) namd_wrapper (int numnodes, file psf_file, file pdb_file, file coord_restart_file, > > >> file velocity_restart_file, file system_restart_file) > > >> { > > >> profile "count" = numnodes; > > >> namd_wrapper @psf_file @pdb_file @coord_restart_file @velocity_restart_file @system_restart_file stdout=@out stderr=@err; > > >> } > > >> > > >> # Range of nodes to test on > > >> int minNodes=1; > > >> int maxNodes=2; > > >> int delta=1; > > >> > > >> # Files > > >> file psf <"h0_solvion.psf">; > > >> file pdb <"h0_solvion.pdb">; > > >> file coord_restart <"h0_eq.0.restart.coor">; > > >> file velocity_restart <"h0_eq.0.restart.vel">; > > >> file system_restart <"h0_eq.0.restart.xsc">; > > >> > > >> foreach nodes in [minNodes:maxNodes:delta] { > > >> file output ; > > >> file error ; > > >> (output, error) = namd_wrapper(nodes, psf, pdb, coord_restart, velocity_restart, system_restart); > > >> } > > >> --- > > >> > > >> In sites.xml, I also set the jobtype to "single" so it doesn't start a worker on each node (namd uses MPI). > > >> > > >> The problem that I'm running into is that, as is, dynamic profiles do not seem to allow you to modify the value for count. I have a workaround for this which involves removing references to "count" in GridExec.java, Execute.java, and TCProfile.java. This works for me in terms of this script, and it works with a few other simple catsn type scripts I've tested. I just wanted to double check to make sure this wouldn't cause any other issues before committing. Here are the changes: > > >> > > >> Index: modules/karajan/src/org/globus/cog/karajan/workflow/nodes/grid/GridExec.java > > >> =================================================================== > > >> --- modules/karajan/src/org/globus/cog/karajan/workflow/nodes/grid/GridExec.java (revision 3472) > > >> +++ modules/karajan/src/org/globus/cog/karajan/workflow/nodes/grid/GridExec.java (working copy) > > >> @@ -56,7 +56,7 @@ > > >> public static final Arg A_STDIN = new Arg.Optional("stdin"); > > >> public static final Arg A_PROVIDER = new Arg.Optional("provider"); > > >> public static final Arg A_SECURITY_CONTEXT = new Arg.Optional("securitycontext"); > > >> - public static final Arg A_COUNT = new Arg.Optional("count"); > > >> + // public static final Arg A_COUNT = new Arg.Optional("count"); > > >> public static final Arg A_HOST_COUNT = new Arg.Optional("hostcount"); > > >> public static final Arg A_JOBTYPE = new Arg.Optional("jobtype"); > > >> public static final Arg A_MAXTIME = new Arg.Optional("maxtime"); > > >> @@ -86,7 +86,8 @@ > > >> static { > > >> setArguments(GridExec.class, new Arg[] { A_EXECUTABLE, A_ARGS, A_ARGUMENTS, A_HOST, > > >> A_STDOUT, A_STDERR, A_STDOUTLOCATION, A_STDERRLOCATION, A_STDIN, A_PROVIDER, > > >> - A_COUNT, A_HOST_COUNT, A_JOBTYPE, A_MAXTIME, A_MAXWALLTIME, A_MAXCPUTIME, > > >> + // A_COUNT, > > >> + A_HOST_COUNT, A_JOBTYPE, A_MAXTIME, A_MAXWALLTIME, A_MAXCPUTIME, > > >> A_ENVIRONMENT, A_QUEUE, A_PROJECT, A_MINMEMORY, A_MAXMEMORY, A_REDIRECT, > > >> A_SECURITY_CONTEXT, A_DIRECTORY, A_NATIVESPEC, A_DELEGATION, A_ATTRIBUTES, > > >> C_ENVIRONMENT, A_FAIL_ON_JOB_ERROR, A_BATCH, C_STAGEIN, C_STAGEOUT, C_CLEANUP, > > >> @@ -346,7 +347,7 @@ > > >> } > > >> } > > >> > > >> - protected final static Arg[] MISC_ATTRS = new Arg[] { A_COUNT, A_HOST_COUNT, A_JOBTYPE, > > >> + protected final static Arg[] MISC_ATTRS = new Arg[] { A_HOST_COUNT, A_JOBTYPE, > > >> A_MAXTIME, A_MAXWALLTIME, A_MAXCPUTIME, A_QUEUE, A_PROJECT, A_MINMEMORY, A_MAXMEMORY }; > > >> > > >> protected void setMiscAttributes(JobSpecification js, VariableStack stack) > > >> > > >> Index: src/org/griphyn/vdl/karajan/lib/TCProfile.java > > >> =================================================================== > > >> --- src/org/griphyn/vdl/karajan/lib/TCProfile.java (revision 5930) > > >> +++ src/org/griphyn/vdl/karajan/lib/TCProfile.java (working copy) > > >> @@ -63,7 +63,6 @@ > > >> > > >> static { > > >> PROFILE_T = new HashMap(); > > >> - PROFILE_T.put("count", GridExec.A_COUNT); > > >> PROFILE_T.put("jobtype", GridExec.A_JOBTYPE); > > >> PROFILE_T.put("maxcputime", GridExec.A_MAXCPUTIME); > > >> PROFILE_T.put("maxmemory", GridExec.A_MAXMEMORY); > > >> > > >> Index: src/org/griphyn/vdl/karajan/lib/Execute.java > > >> =================================================================== > > >> --- src/org/griphyn/vdl/karajan/lib/Execute.java (revision 5930) > > >> +++ src/org/griphyn/vdl/karajan/lib/Execute.java (working copy) > > >> @@ -47,7 +47,7 @@ > > >> static { > > >> setArguments(Execute.class, new Arg[] { A_EXECUTABLE, A_ARGS, A_ARGUMENTS, A_HOST, > > >> A_STDOUT, A_STDERR, A_STDOUTLOCATION, A_STDERRLOCATION, A_STDIN, A_PROVIDER, > > >> - A_COUNT, A_HOST_COUNT, A_JOBTYPE, A_MAXTIME, A_MAXWALLTIME, A_MAXCPUTIME, > > >> + A_HOST_COUNT, A_JOBTYPE, A_MAXTIME, A_MAXWALLTIME, A_MAXCPUTIME, > > >> A_ENVIRONMENT, A_QUEUE, A_PROJECT, A_MINMEMORY, A_MAXMEMORY, A_REDIRECT, > > >> A_SECURITY_CONTEXT, A_DIRECTORY, A_NATIVESPEC, A_DELEGATION, A_ATTRIBUTES, > > >> C_ENVIRONMENT, A_FAIL_ON_JOB_ERROR, A_BATCH, A_REPLICATION_GROUP, > > >> _______________________________________________ > > >> Swift-devel mailing list > > >> Swift-devel at ci.uchicago.edu > > >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > > > > _______________________________________________ > > > Swift-devel mailing list > > > Swift-devel at ci.uchicago.edu > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > > > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel From hategan at mcs.anl.gov Tue Sep 18 23:55:29 2012 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 18 Sep 2012 21:55:29 -0700 Subject: [Swift-devel] Dynamic profiles and count In-Reply-To: <1348028733.21339.0.camel@blabla> References: <1966915315.57505.1347993207886.JavaMail.root@zimbra-mb2.anl.gov> <1348005492.17528.4.camel@blabla> <505930B6.5090401@mcs.anl.gov> <1348026171.20771.0.camel@blabla> <1348028733.21339.0.camel@blabla> Message-ID: <1348030529.21339.3.camel@blabla> For reference: dynamic attributes were passed to execute() through TCProfile(). The latter knew how to properly pass as either arguments or attributes the information from tc.data, but not that from the dynamic attributes. It now properly passes the dynamic attributes as either arguments or attributes. On Tue, 2012-09-18 at 21:25 -0700, Mihael Hategan wrote: > David, > > Can you try r5934? > > Mihael > > On Tue, 2012-09-18 at 20:42 -0700, Mihael Hategan wrote: > > Ah, thanks. I can check and see what's happening. > > > > On Tue, 2012-09-18 at 21:40 -0500, Justin M Wozniak wrote: > > > Just to clarify- this is based on changes already in svn regarding > > > dynamic profiles. That's why the map is there. > > > > > > http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html#_dynamic_profiles > > > > > > David is trying to solve a specific issue with using dynamic profiles > > > and "count": we need to know all about "count" to solve this. > > > > > > This is a paste of my recent email to David about dynamic profiles and > > > issues with "count": > > > > > > Ok, the place to look is in the Swift repo: svn diff -r5206:5207 > > > > > > Apparently, no changes were made to CoG. > > > > > > The dynamic profiles should be visible in the KML. > > > > > > From there, you really need to know your Karajan semantics. The > > > attributes are known to execute2() and thus execute() and Swift's Execute. > > > > > > Now Java. You should be able to see the attributes in GridExec. They > > > are applied to the Task's JobSpecification. These good places to add > > > some trace-level logging. > > > > > > Then, the Task is sent to PBSExecutor. You could add logging here too > > > to make sure the attributes made it. > > > > > > One thing to consider is the possibility that the attribute is being > > > overwritten by some other component. If you add logging to > > > JobSpecification.setAttribute(), you might be able to find that. > > > > > > On 9/18/2012 4:58 PM, Mihael Hategan wrote: > > > > I'm not sure. The way I understand things, your change modifies the way > > > > execute is invoked from: > > > > > > > > execute(executable, ..., count=n) > > > > > > > > to: > > > > > > > > execute(executable, ..., attributes=map(map:entry("count", n), ...)) > > > > > > > > But the semantics are the same. The task object, in both cases, will > > > > have an attribute named "count" equal to n. > > > > > > > > Can you send me the full diff of your changes? > > > > > > > > Mihael > > > > > > > > On Tue, 2012-09-18 at 13:33 -0500, David Kelly wrote: > > > >> Hello, > > > >> > > > >> I have been working on a namd scaling test using Swift. I am using the plain PBS provider at the moment (no coasters). The swift script sets a minimum number of nodes, a maximum number of nodes, iterates through those values, then uses dynamic profiles to change the value of 'count' to modify the number of nodes to request. Here is the script: > > > >> > > > >> --- > > > >> type file; > > > >> > > > >> app (file out, file err) namd_wrapper (int numnodes, file psf_file, file pdb_file, file coord_restart_file, > > > >> file velocity_restart_file, file system_restart_file) > > > >> { > > > >> profile "count" = numnodes; > > > >> namd_wrapper @psf_file @pdb_file @coord_restart_file @velocity_restart_file @system_restart_file stdout=@out stderr=@err; > > > >> } > > > >> > > > >> # Range of nodes to test on > > > >> int minNodes=1; > > > >> int maxNodes=2; > > > >> int delta=1; > > > >> > > > >> # Files > > > >> file psf <"h0_solvion.psf">; > > > >> file pdb <"h0_solvion.pdb">; > > > >> file coord_restart <"h0_eq.0.restart.coor">; > > > >> file velocity_restart <"h0_eq.0.restart.vel">; > > > >> file system_restart <"h0_eq.0.restart.xsc">; > > > >> > > > >> foreach nodes in [minNodes:maxNodes:delta] { > > > >> file output ; > > > >> file error ; > > > >> (output, error) = namd_wrapper(nodes, psf, pdb, coord_restart, velocity_restart, system_restart); > > > >> } > > > >> --- > > > >> > > > >> In sites.xml, I also set the jobtype to "single" so it doesn't start a worker on each node (namd uses MPI). > > > >> > > > >> The problem that I'm running into is that, as is, dynamic profiles do not seem to allow you to modify the value for count. I have a workaround for this which involves removing references to "count" in GridExec.java, Execute.java, and TCProfile.java. This works for me in terms of this script, and it works with a few other simple catsn type scripts I've tested. I just wanted to double check to make sure this wouldn't cause any other issues before committing. Here are the changes: > > > >> > > > >> Index: modules/karajan/src/org/globus/cog/karajan/workflow/nodes/grid/GridExec.java > > > >> =================================================================== > > > >> --- modules/karajan/src/org/globus/cog/karajan/workflow/nodes/grid/GridExec.java (revision 3472) > > > >> +++ modules/karajan/src/org/globus/cog/karajan/workflow/nodes/grid/GridExec.java (working copy) > > > >> @@ -56,7 +56,7 @@ > > > >> public static final Arg A_STDIN = new Arg.Optional("stdin"); > > > >> public static final Arg A_PROVIDER = new Arg.Optional("provider"); > > > >> public static final Arg A_SECURITY_CONTEXT = new Arg.Optional("securitycontext"); > > > >> - public static final Arg A_COUNT = new Arg.Optional("count"); > > > >> + // public static final Arg A_COUNT = new Arg.Optional("count"); > > > >> public static final Arg A_HOST_COUNT = new Arg.Optional("hostcount"); > > > >> public static final Arg A_JOBTYPE = new Arg.Optional("jobtype"); > > > >> public static final Arg A_MAXTIME = new Arg.Optional("maxtime"); > > > >> @@ -86,7 +86,8 @@ > > > >> static { > > > >> setArguments(GridExec.class, new Arg[] { A_EXECUTABLE, A_ARGS, A_ARGUMENTS, A_HOST, > > > >> A_STDOUT, A_STDERR, A_STDOUTLOCATION, A_STDERRLOCATION, A_STDIN, A_PROVIDER, > > > >> - A_COUNT, A_HOST_COUNT, A_JOBTYPE, A_MAXTIME, A_MAXWALLTIME, A_MAXCPUTIME, > > > >> + // A_COUNT, > > > >> + A_HOST_COUNT, A_JOBTYPE, A_MAXTIME, A_MAXWALLTIME, A_MAXCPUTIME, > > > >> A_ENVIRONMENT, A_QUEUE, A_PROJECT, A_MINMEMORY, A_MAXMEMORY, A_REDIRECT, > > > >> A_SECURITY_CONTEXT, A_DIRECTORY, A_NATIVESPEC, A_DELEGATION, A_ATTRIBUTES, > > > >> C_ENVIRONMENT, A_FAIL_ON_JOB_ERROR, A_BATCH, C_STAGEIN, C_STAGEOUT, C_CLEANUP, > > > >> @@ -346,7 +347,7 @@ > > > >> } > > > >> } > > > >> > > > >> - protected final static Arg[] MISC_ATTRS = new Arg[] { A_COUNT, A_HOST_COUNT, A_JOBTYPE, > > > >> + protected final static Arg[] MISC_ATTRS = new Arg[] { A_HOST_COUNT, A_JOBTYPE, > > > >> A_MAXTIME, A_MAXWALLTIME, A_MAXCPUTIME, A_QUEUE, A_PROJECT, A_MINMEMORY, A_MAXMEMORY }; > > > >> > > > >> protected void setMiscAttributes(JobSpecification js, VariableStack stack) > > > >> > > > >> Index: src/org/griphyn/vdl/karajan/lib/TCProfile.java > > > >> =================================================================== > > > >> --- src/org/griphyn/vdl/karajan/lib/TCProfile.java (revision 5930) > > > >> +++ src/org/griphyn/vdl/karajan/lib/TCProfile.java (working copy) > > > >> @@ -63,7 +63,6 @@ > > > >> > > > >> static { > > > >> PROFILE_T = new HashMap(); > > > >> - PROFILE_T.put("count", GridExec.A_COUNT); > > > >> PROFILE_T.put("jobtype", GridExec.A_JOBTYPE); > > > >> PROFILE_T.put("maxcputime", GridExec.A_MAXCPUTIME); > > > >> PROFILE_T.put("maxmemory", GridExec.A_MAXMEMORY); > > > >> > > > >> Index: src/org/griphyn/vdl/karajan/lib/Execute.java > > > >> =================================================================== > > > >> --- src/org/griphyn/vdl/karajan/lib/Execute.java (revision 5930) > > > >> +++ src/org/griphyn/vdl/karajan/lib/Execute.java (working copy) > > > >> @@ -47,7 +47,7 @@ > > > >> static { > > > >> setArguments(Execute.class, new Arg[] { A_EXECUTABLE, A_ARGS, A_ARGUMENTS, A_HOST, > > > >> A_STDOUT, A_STDERR, A_STDOUTLOCATION, A_STDERRLOCATION, A_STDIN, A_PROVIDER, > > > >> - A_COUNT, A_HOST_COUNT, A_JOBTYPE, A_MAXTIME, A_MAXWALLTIME, A_MAXCPUTIME, > > > >> + A_HOST_COUNT, A_JOBTYPE, A_MAXTIME, A_MAXWALLTIME, A_MAXCPUTIME, > > > >> A_ENVIRONMENT, A_QUEUE, A_PROJECT, A_MINMEMORY, A_MAXMEMORY, A_REDIRECT, > > > >> A_SECURITY_CONTEXT, A_DIRECTORY, A_NATIVESPEC, A_DELEGATION, A_ATTRIBUTES, > > > >> C_ENVIRONMENT, A_FAIL_ON_JOB_ERROR, A_BATCH, A_REPLICATION_GROUP, > > > >> _______________________________________________ > > > >> Swift-devel mailing list > > > >> Swift-devel at ci.uchicago.edu > > > >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > > > > > > _______________________________________________ > > > > Swift-devel mailing list > > > > Swift-devel at ci.uchicago.edu > > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > > > > > > > > > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel From davidk at ci.uchicago.edu Wed Sep 19 13:01:25 2012 From: davidk at ci.uchicago.edu (David Kelly) Date: Wed, 19 Sep 2012 13:01:25 -0500 (CDT) Subject: [Swift-devel] Dynamic profiles and count In-Reply-To: <1348028733.21339.0.camel@blabla> Message-ID: <453811726.1958.1348077685393.JavaMail.root@zimbra-mb2.anl.gov> That seems to be working great. Thanks! David ----- Original Message ----- > From: "Mihael Hategan" > To: "Justin M Wozniak" > Cc: "swift-devel Devel" > Sent: Tuesday, September 18, 2012 11:25:33 PM > Subject: Re: [Swift-devel] Dynamic profiles and count > David, > > Can you try r5934? > > Mihael > > On Tue, 2012-09-18 at 20:42 -0700, Mihael Hategan wrote: > > Ah, thanks. I can check and see what's happening. > > > > On Tue, 2012-09-18 at 21:40 -0500, Justin M Wozniak wrote: > > > Just to clarify- this is based on changes already in svn regarding > > > dynamic profiles. That's why the map is there. > > > > > > http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html#_dynamic_profiles > > > > > > David is trying to solve a specific issue with using dynamic > > > profiles > > > and "count": we need to know all about "count" to solve this. > > > > > > This is a paste of my recent email to David about dynamic profiles > > > and > > > issues with "count": > > > > > > Ok, the place to look is in the Swift repo: svn diff -r5206:5207 > > > > > > Apparently, no changes were made to CoG. > > > > > > The dynamic profiles should be visible in the KML. > > > > > > From there, you really need to know your Karajan semantics. The > > > attributes are known to execute2() and thus execute() and Swift's > > > Execute. > > > > > > Now Java. You should be able to see the attributes in GridExec. > > > They > > > are applied to the Task's JobSpecification. These good places to > > > add > > > some trace-level logging. > > > > > > Then, the Task is sent to PBSExecutor. You could add logging here > > > too > > > to make sure the attributes made it. > > > > > > One thing to consider is the possibility that the attribute is > > > being > > > overwritten by some other component. If you add logging to > > > JobSpecification.setAttribute(), you might be able to find that. > > > > > > On 9/18/2012 4:58 PM, Mihael Hategan wrote: > > > > I'm not sure. The way I understand things, your change modifies > > > > the way > > > > execute is invoked from: > > > > > > > > execute(executable, ..., count=n) > > > > > > > > to: > > > > > > > > execute(executable, ..., attributes=map(map:entry("count", n), > > > > ...)) > > > > > > > > But the semantics are the same. The task object, in both cases, > > > > will > > > > have an attribute named "count" equal to n. > > > > > > > > Can you send me the full diff of your changes? > > > > > > > > Mihael > > > > > > > > On Tue, 2012-09-18 at 13:33 -0500, David Kelly wrote: > > > >> Hello, > > > >> > > > >> I have been working on a namd scaling test using Swift. I am > > > >> using the plain PBS provider at the moment (no coasters). The > > > >> swift script sets a minimum number of nodes, a maximum number > > > >> of nodes, iterates through those values, then uses dynamic > > > >> profiles to change the value of 'count' to modify the number of > > > >> nodes to request. Here is the script: > > > >> > > > >> --- > > > >> type file; > > > >> > > > >> app (file out, file err) namd_wrapper (int numnodes, file > > > >> psf_file, file pdb_file, file coord_restart_file, > > > >> file > > > >> velocity_restart_file, > > > >> file > > > >> system_restart_file) > > > >> { > > > >> profile "count" = numnodes; > > > >> namd_wrapper @psf_file @pdb_file @coord_restart_file > > > >> @velocity_restart_file @system_restart_file stdout=@out > > > >> stderr=@err; > > > >> } > > > >> > > > >> # Range of nodes to test on > > > >> int minNodes=1; > > > >> int maxNodes=2; > > > >> int delta=1; > > > >> > > > >> # Files > > > >> file psf <"h0_solvion.psf">; > > > >> file pdb <"h0_solvion.pdb">; > > > >> file coord_restart <"h0_eq.0.restart.coor">; > > > >> file velocity_restart <"h0_eq.0.restart.vel">; > > > >> file system_restart <"h0_eq.0.restart.xsc">; > > > >> > > > >> foreach nodes in [minNodes:maxNodes:delta] { > > > >> file output > > >> file=@strcat("logs/scaling-", nodes, ".out.txt")>; > > > >> file error > > >> file=@strcat("logs/scaling-", nodes, ".err.txt")>; > > > >> (output, error) = namd_wrapper(nodes, psf, pdb, > > > >> coord_restart, velocity_restart, system_restart); > > > >> } > > > >> --- > > > >> > > > >> In sites.xml, I also set the jobtype to "single" so it doesn't > > > >> start a worker on each node (namd uses MPI). > > > >> > > > >> The problem that I'm running into is that, as is, dynamic > > > >> profiles do not seem to allow you to modify the value for > > > >> count. I have a workaround for this which involves removing > > > >> references to "count" in GridExec.java, Execute.java, and > > > >> TCProfile.java. This works for me in terms of this script, and > > > >> it works with a few other simple catsn type scripts I've > > > >> tested. I just wanted to double check to make sure this > > > >> wouldn't cause any other issues before committing. Here are the > > > >> changes: > > > >> > > > >> Index: > > > >> modules/karajan/src/org/globus/cog/karajan/workflow/nodes/grid/GridExec.java > > > >> =================================================================== > > > >> --- > > > >> modules/karajan/src/org/globus/cog/karajan/workflow/nodes/grid/GridExec.java > > > >> (revision 3472) > > > >> +++ > > > >> modules/karajan/src/org/globus/cog/karajan/workflow/nodes/grid/GridExec.java > > > >> (working copy) > > > >> @@ -56,7 +56,7 @@ > > > >> public static final Arg A_STDIN = new > > > >> Arg.Optional("stdin"); > > > >> public static final Arg A_PROVIDER = new > > > >> Arg.Optional("provider"); > > > >> public static final Arg A_SECURITY_CONTEXT = new > > > >> Arg.Optional("securitycontext"); > > > >> - public static final Arg A_COUNT = new Arg.Optional("count"); > > > >> + // public static final Arg A_COUNT = new > > > >> Arg.Optional("count"); > > > >> public static final Arg A_HOST_COUNT = new > > > >> Arg.Optional("hostcount"); > > > >> public static final Arg A_JOBTYPE = new > > > >> Arg.Optional("jobtype"); > > > >> public static final Arg A_MAXTIME = new > > > >> Arg.Optional("maxtime"); > > > >> @@ -86,7 +86,8 @@ > > > >> static { > > > >> setArguments(GridExec.class, new Arg[] { > > > >> A_EXECUTABLE, A_ARGS, A_ARGUMENTS, A_HOST, > > > >> A_STDOUT, A_STDERR, > > > >> A_STDOUTLOCATION, > > > >> A_STDERRLOCATION, A_STDIN, > > > >> A_PROVIDER, > > > >> - A_COUNT, A_HOST_COUNT, A_JOBTYPE, A_MAXTIME, A_MAXWALLTIME, > > > >> A_MAXCPUTIME, > > > >> + // A_COUNT, > > > >> + A_HOST_COUNT, A_JOBTYPE, A_MAXTIME, A_MAXWALLTIME, > > > >> A_MAXCPUTIME, > > > >> A_ENVIRONMENT, A_QUEUE, > > > >> A_PROJECT, A_MINMEMORY, > > > >> A_MAXMEMORY, A_REDIRECT, > > > >> A_SECURITY_CONTEXT, > > > >> A_DIRECTORY, A_NATIVESPEC, > > > >> A_DELEGATION, A_ATTRIBUTES, > > > >> C_ENVIRONMENT, > > > >> A_FAIL_ON_JOB_ERROR, A_BATCH, > > > >> C_STAGEIN, C_STAGEOUT, > > > >> C_CLEANUP, > > > >> @@ -346,7 +347,7 @@ > > > >> } > > > >> } > > > >> > > > >> - protected final static Arg[] MISC_ATTRS = new Arg[] { > > > >> A_COUNT, A_HOST_COUNT, A_JOBTYPE, > > > >> + protected final static Arg[] MISC_ATTRS = new Arg[] { > > > >> A_HOST_COUNT, A_JOBTYPE, > > > >> A_MAXTIME, A_MAXWALLTIME, > > > >> A_MAXCPUTIME, A_QUEUE, A_PROJECT, > > > >> A_MINMEMORY, A_MAXMEMORY }; > > > >> > > > >> protected void setMiscAttributes(JobSpecification js, > > > >> VariableStack stack) > > > >> > > > >> Index: src/org/griphyn/vdl/karajan/lib/TCProfile.java > > > >> =================================================================== > > > >> --- src/org/griphyn/vdl/karajan/lib/TCProfile.java (revision > > > >> 5930) > > > >> +++ src/org/griphyn/vdl/karajan/lib/TCProfile.java (working > > > >> copy) > > > >> @@ -63,7 +63,6 @@ > > > >> > > > >> static { > > > >> PROFILE_T = new HashMap(); > > > >> - PROFILE_T.put("count", GridExec.A_COUNT); > > > >> PROFILE_T.put("jobtype", GridExec.A_JOBTYPE); > > > >> PROFILE_T.put("maxcputime", > > > >> GridExec.A_MAXCPUTIME); > > > >> PROFILE_T.put("maxmemory", > > > >> GridExec.A_MAXMEMORY); > > > >> > > > >> Index: src/org/griphyn/vdl/karajan/lib/Execute.java > > > >> =================================================================== > > > >> --- src/org/griphyn/vdl/karajan/lib/Execute.java (revision > > > >> 5930) > > > >> +++ src/org/griphyn/vdl/karajan/lib/Execute.java (working copy) > > > >> @@ -47,7 +47,7 @@ > > > >> static { > > > >> setArguments(Execute.class, new Arg[] { > > > >> A_EXECUTABLE, A_ARGS, A_ARGUMENTS, A_HOST, > > > >> A_STDOUT, A_STDERR, > > > >> A_STDOUTLOCATION, > > > >> A_STDERRLOCATION, A_STDIN, > > > >> A_PROVIDER, > > > >> - A_COUNT, A_HOST_COUNT, A_JOBTYPE, A_MAXTIME, A_MAXWALLTIME, > > > >> A_MAXCPUTIME, > > > >> + A_HOST_COUNT, A_JOBTYPE, A_MAXTIME, A_MAXWALLTIME, > > > >> A_MAXCPUTIME, > > > >> A_ENVIRONMENT, A_QUEUE, > > > >> A_PROJECT, A_MINMEMORY, > > > >> A_MAXMEMORY, A_REDIRECT, > > > >> A_SECURITY_CONTEXT, > > > >> A_DIRECTORY, A_NATIVESPEC, > > > >> A_DELEGATION, A_ATTRIBUTES, > > > >> C_ENVIRONMENT, > > > >> A_FAIL_ON_JOB_ERROR, A_BATCH, > > > >> A_REPLICATION_GROUP, > > > >> _______________________________________________ > > > >> Swift-devel mailing list > > > >> Swift-devel at ci.uchicago.edu > > > >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > > > > > > _______________________________________________ > > > > Swift-devel mailing list > > > > Swift-devel at ci.uchicago.edu > > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > > > > > > > > > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel