From iraicu at cs.iit.edu Thu Aug 2 09:51:55 2012 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Thu, 02 Aug 2012 09:51:55 -0500 Subject: [Swift-user] CFP: 5th Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS) 2012 -- at IEEE/ACM Supercomputing 2012 Message-ID: <501A940B.70001@cs.iit.edu> Call for Papers --------------------------------------------------------------------------------------- The 5th Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS) 2012 http://datasys.cs.iit.edu/events/MTAGS12/ --------------------------------------------------------------------------------------- November 12th, 2012 Salt Lake City, Utah, USA Co-located with with IEEE/ACM International Conference for High Performance Computing, Networking, Storage and Analysis (SC12) ======================================================================================= The 5th workshop on Many-Task Computing on Grids and Supercomputers (MTAGS) will provide the scientific community a dedicated forum for presenting new research, development, and deployment efforts of large-scale many-task computing (MTC) applications on large scale clusters, Grids, Supercomputers, and Cloud Computing infrastructure. MTC, the theme of the workshop encompasses loosely coupled applications, which are generally composed of many tasks (both independent and dependent tasks) to achieve some larger application goal. This workshop will cover challenges that can hamper efficiency and utilization in running applications on large-scale systems, such as local resource manager scalability and granularity, efficient utilization of raw hardware, parallel file system contention and scalability, data management, I/O management, reliability at scale, and application scalability. We welcome paper submissions on all theoretical, simulations, and systems topics related to MTC, but we give special consideration to papers addressing petascale to exascale challenges. Papers will be peer-reviewed, and accepted papers will be published in the workshop proceedings as part of the IEEE digital library (pending approval). The workshop will be co-located with the IEEE/ACM Supercomputing 2012 Conference in Salt Lake City Utah on November 12th, 2012. For more information, please see http://datasys.cs.iit.edu/events/MTAGS12/. For more information on past workshops, please see MTAGS11, MTAGS10, MTAGS09, and MTAGS08. We also ran a Special Issue on Many-Task Computing in the IEEE Transactions on Parallel and Distributed Systems (TPDS) which has appeared in June 2011; the proceedings can be found online at http://www.computer.org/portal/web/csdl/abs/trans/td/2011/06/ttd201106toc.htm. We, the workshop organizers, also published a highly relevant paper that defines Many-Task Computing which was published in MTAGS08, titled "Many-Task Computing for Grids and Supercomputers" (http://www.cs.iit.edu/~iraicu/research/publications/2008_MTAGS08_MTC.pdf); we encourage potential authors to read this paper. Topics --------------------------------------------------------------------------------------- We invite the submission of original work that is related to the topics below. The papers can be either short (4 pages) position papers, or long (8 pages) research papers. Topics of interest include (in the context of Many-Task Computing): * Compute Resource Management * Scheduling * Job execution frameworks * Local resource manager extensions * Performance evaluation of resource managers in use on large scale systems * Dynamic resource provisioning * Techniques to manage many-core resources and/or GPUs * Challenges and opportunities in running many-task workloads on HPC systems * Challenges and opportunities in running many-task workloads on Cloud Computing infrastructure * Storage architectures and implementations * Distributed file systems * Parallel file systems * Distributed meta-data management * Content distribution systems for large data * Data caching frameworks and techniques * Data management within and across data centers * Data-aware scheduling * Data-intensive computing applications * Eventual-consistency storage usage and management * Programming models and tools * Map-reduce and its generalizations * Many-task computing middleware and applications * Parallel programming frameworks * Ensemble MPI techniques and frameworks * Service-oriented science applications * Large-Scale Workflow Systems * Workflow system performance and scalability analysis * Scalability of workflow systems * Workflow infrastructure and e-Science middleware * Programming Paradigms and Models * Large-Scale Many-Task Applications * High-throughput computing (HTC) applications * Data-intensive applications * Quasi-supercomputing applications, deployments, and experiences * Performance Evaluation * Performance evaluation * Real systems * Simulations * Reliability of large systems * How MTC Addresses Challenges of Petascale and Exascale Computing * Concurrency & Programmability * I/O & Memory * Energy * Resilience * Heterogeneity Paper Submission and Publication --------------------------------------------------------------------------------------- Authors are invited to submit papers with unpublished, original work of not more than 8 pages of double column text using single spaced 10 point size on 8.5 x 11 inch pages, as per IEEE 8.5 x 11 manuscript guidelines; document templates can be found at http://www.ieee.org/conferences_events/conferences/publishing/templates.html. We are also seeking position papers of no more than 4 pages in length. The final 4/8 page papers in PDF format must be submitted online at https://cmt.research.microsoft.com/MTAGS2012/ before the deadline. Papers will be peer-reviewed, and accepted papers will be published in the workshop proceedings as part of the IEEE digital library (pending approval). Notifications of the paper decisions will be sent out by October 12th, 2012. Selected excellent work may be eligible for additional post-conference publication as journal articles or book chapters, such as the previous Special Issue on Many-Task Computing in the IEEE Transactions on Parallel and Distributed Systems (TPDS) which has appeared in June 2011. Submission implies the willingness of at least one of the authors to register and present the paper. For more information, please http://datasys.cs.iit.edu/events/MTAGS12/, or send email to mtags12-chairs at datasys.cs.iit.edu. Important Dates --------------------------------------------------------------------------------------- * Abstract submission: September 10th, 2012 (11:59PM PST) * Paper submission: September 17th, 2012 (11:59PM PST) * Acceptance notification: October 12th, 2012 * Final papers due: November 7th, 2012 Committee Members --------------------------------------------------------------------------------------- Workshop Chairs (mtags12-chairs at datasys.cs.iit.edu) * Ioan Raicu, Illinois Institute of Technology & Argonne National Laboratory * Ian Foster, University of Chicago & Argonne National Laboratory * Yong Zhao, University of Electronic Science and Technology of China Steering Committee * David Abramson, Monash University, Australia * Jack Dongara, University of Tennessee, USA * Geoffrey Fox, Indiana University, USA * Manish Parashar, Rutgers University, USA * Marc Snir, University of Illinois at Urbana Champaign, USA * Xian-He Sun, Illinois Institute of Technology, USA * Weimin Zheng, Tsinghua University, China Publicity Chair (mtags12-publicity at datasys.cs.iit.edu) * Zhao Zhang, University of Chicago, USA Program Committee Chair (mtags12-pc-chair at datasys.cs.iit.edu) * Justin Wozniak, Argonne National Laboratory, USA Technical Committee * Roger Barga, Microsoft Research, USA * Kyle Chard, University of Chicago, USA * Dennis Gannon, Microsoft Research, USA * Florin Isaila, Universidad Carlos III de Madrid, Spain * Hui Jin, Oracle Corporation, USA * Daniel S. Katz, University of Chicago, USA * Zhiling Lan, Illinois Institute of Technology, USA * Mike Lang, Los Alamos National Laboratory, USA * Gregor von Laszewski, Indiana University, USA * Reagan Moore, University of North Carolina, Chappel Hill, USA * David O'Hallaron, Carnegie Mellon University, Intel Labs, USA * Marlon Pierce, Indiana University, USA * Judy Qiu, Indiana University, USA * Lavanya Ramakrishnan, Lawrence Berkeley National Laboratory, USA * Kui Ren, SUNY Buffalo, USA * Matei Ripeanu, University of British Columbia, Canada * Wei Tang, Argonne National Laboratory, USA * Valerie Taylor, Texas A&M, USA * Ken Yocum, University of California, San Diego, USA * Zhifeng Yun, Louisiana State University, USA * Zhao Zhang, University of Chicago, USA -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ ================================================================= ================================================================= From turam at mcs.anl.gov Thu Aug 2 11:23:30 2012 From: turam at mcs.anl.gov (Thomas Uram) Date: Thu, 2 Aug 2012 11:23:30 -0500 Subject: [Swift-user] How to determine length of an array? Message-ID: <06C7A6B3-A777-4099-A6EA-CE02D5199609@mcs.anl.gov> Some functionality of my script is conditional based on the contents of an array, so I need to determine the array length. The path described by the documentation leads to an exception. How should I determine array length? Or, a lesser requirement: how can I determine if an array is empty? The user guide (http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html) says: @length(array) will return the length of an array in Swift. This function will wait for all elements in the array to be written before returning the length. But the following Swift script: type file; file input_files[] ; trace( @length( input_files ) ) ; Produces this output: Swift 0.93 swift-r5483 cog-r3339 RunID: 20120802-1218-5dfj571d (input): found 255 files Progress: time: Thu, 02 Aug 2012 12:18:51 -0400 Execution failed: java.lang.ClassCastException: org.griphyn.vdl.mapping.RootArrayDataNode cannot be cast to java.util.Map at org.griphyn.vdl.karajan.lib.swiftscript.Misc.swiftscript_length(Misc.java:399) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.globus.cog.karajan.workflow.nodes.functions.FunctionsCollection.function(FunctionsCollection.java:82) at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:27) at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194) at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28) at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29) at org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20) at org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63) at org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139) at org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197) at org.globus.cog.karajan.workflow.FlowElementWrapper.start(FlowElementWrapper.java:227) at org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:104) at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) -------------- next part -------------- An HTML attachment was scrubbed... URL: From wilde at mcs.anl.gov Thu Aug 2 12:29:00 2012 From: wilde at mcs.anl.gov (Michael Wilde) Date: Thu, 2 Aug 2012 12:29:00 -0500 (CDT) Subject: [Swift-user] How to determine length of an array? In-Reply-To: <06C7A6B3-A777-4099-A6EA-CE02D5199609@mcs.anl.gov> Message-ID: <226809073.47983.1343928540955.JavaMail.root@zimbra.anl.gov> Hi Tom, This should work. Its hopefully fixed in trunk, but if not, we should fix it there, quickly. In the meantime, for moderate sized arrays you may be able to use the hack below as a temporary solution. - Mike --- type file; app (file o) countArgs (int v[]) { sh "-c" "echo $#" v stdout=@o; } (int s) sizeOfFileArray (file fa[]) { int ia[]; foreach f,i in fa { ia[i] = 1; } s = readData(countArgs(ia)); } file input_files[] ; # trace( @length( input_files ) ) ; trace("size", sizeOfFileArray(input_files)); ----- Original Message ----- > From: "Thomas Uram" > To: "swift user" > Sent: Thursday, August 2, 2012 11:23:30 AM > Subject: [Swift-user] How to determine length of an array? > Some functionality of my script is conditional based on the contents > of an array, so I need to determine the array length. The path > described by the documentation leads to an exception. How should I > determine array length? Or, a lesser requirement: how can I determine > if an array is empty? > > > The user guide ( > http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html > ) says: > > > @length(array) will return the length of an array in Swift. This > function will wait for all elements in the array to be written before > returning the length. > > > But the following Swift script: > > > > type file; > file input_files[] ; > trace( @length( input_files ) ) ; > > > Produces this output: > > > > Swift 0.93 swift-r5483 cog-r3339 > > > RunID: 20120802-1218-5dfj571d > (input): found 255 files > Progress: time: Thu, 02 Aug 2012 12:18:51 -0400 > Execution failed: > java.lang.ClassCastException: > org.griphyn.vdl.mapping.RootArrayDataNode cannot be cast to > java.util.Map > at > org.griphyn.vdl.karajan.lib.swiftscript.Misc.swiftscript_length(Misc.java:399) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.globus.cog.karajan.workflow.nodes.functions.FunctionsCollection.function(FunctionsCollection.java:82) > at > org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:27) > at > org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) > at > org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) > at > org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28) > at > org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29) > at > org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20) > at > org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197) > at > org.globus.cog.karajan.workflow.FlowElementWrapper.start(FlowElementWrapper.java:227) > at > org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:104) > at > org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory From wilde at mcs.anl.gov Thu Aug 2 12:53:41 2012 From: wilde at mcs.anl.gov (Michael Wilde) Date: Thu, 2 Aug 2012 12:53:41 -0500 (CDT) Subject: [Swift-user] How to determine length of an array? In-Reply-To: <226809073.47983.1343928540955.JavaMail.root@zimbra.anl.gov> Message-ID: <1080976903.48034.1343930021381.JavaMail.root@zimbra.anl.gov> I cleaned this example up a bit and also tested @length() in trunk. Both seem to work. The revised example is below. - Mike $ cat size.swift type file; app (file o) countargs (int v[]) { sh "-c" "echo $#" "" v stdout=@o; } (int s) sizeof (file fa[]) { int ia[]; foreach f,i in fa { ia[i] = 1; } s = readData(countargs(ia)); } file input_files[] ; trace("length", @length( input_files ) ) ; trace("size", sizeof(input_files)); $ ls -1 *.txt | wc -l 24 $ swift -tc.file tc size.swift no sites file specified, setting to default: /home/wilde/swift/rev/trunk/etc/sites.xml Warning: Procedure countargs is deprecated, at line 14 Warning: Procedure sizeof is deprecated, at line 21 Swift trunk swift-r5781 cog-r3368 (cog modified locally) RunID: 20120802-1251-h7fvjrv7 (input): found 24 files Progress: time: Thu, 02 Aug 2012 12:51:16 -0500 SwiftScript trace: length, 24 SwiftScript trace: size, 24 Final status: Thu, 02 Aug 2012 12:51:16 -0500 Finished successfully:1 $ ----- Original Message ----- > From: "Michael Wilde" > To: "Thomas Uram" > Cc: "swift user" > Sent: Thursday, August 2, 2012 12:29:00 PM > Subject: Re: [Swift-user] How to determine length of an array? > Hi Tom, > > This should work. Its hopefully fixed in trunk, but if not, we should > fix it there, quickly. > > In the meantime, for moderate sized arrays you may be able to use the > hack below as a temporary solution. > > - Mike > > --- > > type file; > > > app (file o) countArgs (int v[]) > { > sh "-c" "echo $#" v stdout=@o; > } > > (int s) sizeOfFileArray (file fa[]) > { > int ia[]; > foreach f,i in fa { > ia[i] = 1; > } > s = readData(countArgs(ia)); > } > > file input_files[] ; > > # trace( @length( input_files ) ) ; > > trace("size", sizeOfFileArray(input_files)); > > > > ----- Original Message ----- > > From: "Thomas Uram" > > To: "swift user" > > Sent: Thursday, August 2, 2012 11:23:30 AM > > Subject: [Swift-user] How to determine length of an array? > > Some functionality of my script is conditional based on the contents > > of an array, so I need to determine the array length. The path > > described by the documentation leads to an exception. How should I > > determine array length? Or, a lesser requirement: how can I > > determine > > if an array is empty? > > > > > > The user guide ( > > http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html > > ) says: > > > > > > @length(array) will return the length of an array in Swift. This > > function will wait for all elements in the array to be written > > before > > returning the length. > > > > > > But the following Swift script: > > > > > > > > type file; > > file input_files[] ; > > trace( @length( input_files ) ) ; > > > > > > Produces this output: > > > > > > > > Swift 0.93 swift-r5483 cog-r3339 > > > > > > RunID: 20120802-1218-5dfj571d > > (input): found 255 files > > Progress: time: Thu, 02 Aug 2012 12:18:51 -0400 > > Execution failed: > > java.lang.ClassCastException: > > org.griphyn.vdl.mapping.RootArrayDataNode cannot be cast to > > java.util.Map > > at > > org.griphyn.vdl.karajan.lib.swiftscript.Misc.swiftscript_length(Misc.java:399) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at > > org.globus.cog.karajan.workflow.nodes.functions.FunctionsCollection.function(FunctionsCollection.java:82) > > at > > org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:27) > > at > > org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194) > > at > > org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) > > at > > org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) > > at > > org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28) > > at > > org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29) > > at > > org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20) > > at > > org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63) > > at > > org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139) > > at > > org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197) > > at > > org.globus.cog.karajan.workflow.FlowElementWrapper.start(FlowElementWrapper.java:227) > > at > > org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:104) > > at > > org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > > at > > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:619) > > > > > > _______________________________________________ > > Swift-user mailing list > > Swift-user at ci.uchicago.edu > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > -- > Michael Wilde > Computation Institute, University of Chicago > Mathematics and Computer Science Division > Argonne National Laboratory > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory From iraicu at cs.iit.edu Tue Aug 7 22:47:07 2012 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Tue, 07 Aug 2012 22:47:07 -0500 Subject: [Swift-user] Call for Participation: The 9th ACM International Conference on Autonomic Computing (ICAC 2012) -- September 17-21, San Jose CA Message-ID: <5021E13B.108@cs.iit.edu> ********************************************************************** CALL FOR PARTICIPATION ====================== The 9th ACM International Conference on Autonomic Computing (ICAC 2012) San Jose, California, USA September 17-21, 2012 http://icac2012.cs.fiu.edu Sponsored by ACM ********************************************************************** Online registration is open at http://icac2012.cs.fiu.edu/registration.shtm Reduced fees are available for those registering by September 2, 2012. This year's technical program features: - 3 distinguished keynote speakers: * Dr. Amin Vahdat, Google/UCSD * Dr. Subutai Ahmad (VP Engineering), Numenta * Dr. Eitan Frachtenberg, Facebook - 24 outstanding technical papers (15 full + 9 short): * covering core and emerging topics such as clouds, virtualization, control, monitoring and diagnosis, and energy * Half of the papers involve authors from industry or government labs - 4 co-located workshops covering hot topics in: * Feedback Computing * Self-Aware Internet of Things * Management of Big Data Systems * Federated Clouds The Conference will be held at the Fairmont Hotel in downtown San Jose, CA, USA. ********************************************************************** IMPORTANT DATES =============== Early registration deadline: September 2, 2012 Hotel special rate deadline: September 7, 2012 ********************************************************************** CORPORATE SPONSORS ================== Gold Level Partner: IBM Conference partners: VMware, HP, Neustar PhD Student Sponsor: Google Other Sponsor: NEC Labs ********************************************************************** PRELIMINARY PROGRAM =================== ====================================================================== MONDAY, SEPTEMBER 17, 2012 - WORKSHOPS Feedback Computing 2012 Self-Aware Internet of Things 2012 ====================================================================== TUESDAY, SEPTEMBER 18, 2012 - MAIN CONFERENCE 8:00AM ? 8:45AM Registration 8:45AM ? 9:00AM Welcome Remarks 9:00AM ? 10:00AM Keynote Talk I Symbiosis in Scale Out Networking and Data Management Amin Vahdat, Google and University of California, San Diego 10:00AM ? 10.30AM Break 10:30AM ? 12:00PM Session : Virtualization Net-Cohort: Detecting and Managing VM Ensembles in Virtualized Data Centers Liting Hu, Karsten Schwan (Georgia Institute of Technology); Ajay Gulati (VMware); Junjie Zhang, Chengwei Wang (Georgia Institute of Technology) Application-aware Cross-layer Virtual Machine Resource Management Lixi Wang, Jing Xu, Ming Zhao (Florida International University) Shifting GEARS to Enable Guest-context Virtual Services Kyle Hale, Lei Xia, Peter Dinda (Northwestern University) 12:00PM-1:30PM Lunch ---------------------------------------------------------------------- 1:30PM - 3:30PM Session: Performance and Resource Management When Average is Not Average: Large Response Time Fluctuations in N-tier Systems Qingyang Wang (Georgia Institute of Technology); Yasuhiko Kanemasa, Motoyuki Kawaba (Fujitsu Laboratories Ltd.); Calton Pu (Georgia Institute of Technology) Provisioning Multi-tier Cloud Applications Using Statistical Bounds on Sojourn Time Upendra Sharma, Prashant Shenoy, Don Towsley (University of Massachusetts Amherst) Automated Profiling and Resource Management of Pig Programs for Meeting Service Level Objectives Zhuoyao Zhang (University of Pennsylvania); Ludmila Cherkasova (Hewlett-Packard Labs); Abhishek Verma (University of Illinois at Urbana-Champaign); Boon Thau Loo (University of Pennsylvania) AROMA: Automated Resource Allocation and Configuration of MapReduce Environment in the Cloud Palden Lama, Xiaobo Zhou (University of Colorado at Colorado Springs) 3:30PM-4:00PM Break 4:00PM ? 5:15PM Short Papers I Locomotion at Location: When the Rubber hits the Road Gerold Hoelzl, Marc Kurz, Alois Ferscha (Johannes Kepler University Linz, Austria) An Autonomic Resource Provisioning Framework for Mobile Computing Grids Hariharasudhan Viswanathan, Eun Kyung Lee, Ivan Rodero, Dario Pompili (Rutgers University) A Self-Tuning Self-Optimizing Approach for Automated Network Anomaly Detection Systems Dennis Ippoliti, Xiaobo Zhou (University of Colorado at Colorado Springs) Offline and On-Demand Event Correlation for Operations Management of Large Scale IT Systems Chetan Gupta (Hewlett-Packard Labs) PowerTracer: Tracing Requests in Multi-tier Services to Diagnose Energy Inefficiency Gang Lu, Jianfeng Zhan (Institute of Computing Technology, Chinese Academy of Sciences); Haining Wang (College of William and Mary); Lin Yuan (Institute of Computing Technology, Chinese Academy of Sciences); Chuliang Weng (Shanghai Jiao Tong University, China) 6:00PM - 9:00PM Conference Dinner ====================================================================== WEDNESDAY, SEPTEMBER 19 - MAIN CONFERENCE 8:00AM ? 9:00AM Registration 9:00AM ? 10:00AM Keynote Talk II Automated Machine Learning For Autonomic Computing Subutai Ahmad, VP Engineering, Numenta 10:00AM - 10:30AM Break 10:30AM ? 12:00PM Session: Control-Based Approaches Budget-based Control for Interactive Services with Adaptive Execution Yuxiong He, Zihao Ye, Qiang Fu, Sameh Elnikety (Microsoft Research) On the Design of Decentralized Control Architectures for Workload Consolidation in Large-Scale Server Clusters Rui Wang, Nagarajan Kandasamy (Drexel University) Transactional Auto Scaler: Elastic Scaling of In-Memory Transactional Data Grids Diego Didona, Paolo Romano (Instituto Superior T?cnico/INESC-ID); Sebastiano Peluso, Francesco Quaglia (Sapienza, Universit? di Roma) 12:00PM - 1:30PM Lunch ---------------------------------------------------------------------- 1:30PM - 2:30PM Session: Energy Adaptive Green Hosting Nan Deng, Christopher Stewart, Jaimie Kelley (The Ohio State University); Daniel Gmach, Martin Arlitt (Hewlett Packard Labs) Dynamic Energy-Aware Capacity Provisioning for Cloud Computing Environments Qi Zhang, Mohamed Faten Zhani (University of Waterloo); Shuo Zhang (National University of Defense Technology); Quanyan Zhu (University of Illinois at Urbana-Champaign); Raouf Boutaba (University of Waterloo); Joseph L. Hellerstein (Google, Inc.) 2:30PM - 3:30PM Short Papers II VESPA: Multi-Layered Self-Protection for Cloud Resources Aur?lien Wailly, Marc Lacoste (Orange Labs); Herv? Debar (T?l?com SudParis) Usage Patterns in Multi-tenant Data Centers: a Temporal Perspective Robert Birke, Lydia Y. Chen (IBM Research Zurich Lab); Evgenia Smirni (College of William and Mary) Toward Fast Eventual Consistency with Performance Guarantees Feng Yan (College of William and Mary); Alma Riska (EMC Corporation); Evgenia Smirni (College of William and Mary) Optimal Autoscaling in the IaaS Cloud Hamoun Ghanbari, Bradley Simmons, Marin Litoiu, Cornel Barna (York University); Gabriel Iszlai (IBM Toronto) 3:30PM ? 4:00PM Break 4:00PM - 6:00PM Poster and Demo Session 6:00PM-9:00PM Conference Outing (tentative) ====================================================================== THURSDAY, SEPTEMBER 20, 2012 - MAIN CONFERENCE 8:00AM ? 9:00AM Registration 9:00AM ? 10:00AM Keynote Talk III High Efficiency at Web Scale Eitan Frachtenberg, Facebook 10:00AM-10:30AM Break 10:30AM - 12:00PM Session: Diagnosis and Monitoring Chair: TBD 3-Dimensional Root Cause Diagnosis via Co-analysi Ziming Zheng, Li Yu, Zhiling Lan (Illinois Institute of Technology); Terry Jones (Oak Ridge National Laboratory) UBL: Unsupervised Behavior Learning for Predicting Performance Anomalies in Virtualized Cloud Systems Daniel J. Dean, Hiep Nguyen, Xiaohui Gu (North Carolina State University) Evaluating Compressive Sampling Strategies for Performance Monitoring of Data Centers Tingshan Huang, Nagarajan Kandasamy, Harish Sethu (Drexel University) 12:00PM Adjourn ====================================================================== FRIDAY, SEPTEMBER 21, 2012 - WORKSHOPS Management of Big Data Systems 2012 Federated Clouds 2012 ********************************************************************** ORGANIZERS ========== GENERAL CHAIR Dejan Milojicic, HP Labs PROGRAM CHAIRS Dongyan Xu, Purdue University Vanish Talwar, HP Labs INDUSTRY CHAIR Xiaoyun Zhu, VMware WORKSHOPS CHAIR Fred Douglis, EMC POSTERS/DEMO/EXHIBITS CHAIR Eno Thereska, Microsoft Research FINANCE CHAIR Michael Kozuch, Intel LOCAL ARRANGEMENT CHAIR Jessica Blaine PUBLICITY CHAIRS Daniel Batista, University of S?o Paulo Vartan Padaryan, ISP/Russian Academy of Sci. Ioan Raicu, Illinois Inst. of Technology Jianfeng Zhan, ICT/Chinese Academy of Sci. Ming Zhao, Florida Intl. University PROGRAM COMMITTEE Tarek Abdelzaher, UIUC Umesh Bellur, IIT, Bombay Ken Birman, Cornell University Rajkumar Buyya, Univ. of Melbourne Rocky Chang, Hong Kong Polytechnic University Yuan Chen, HP Labs Alva Couch, Tufts University Peter Dinda, Northwestern University Fred Douglis, EMC Renato Figueiredo, University of Florida Mohamed Hefeeda, QCRI Joe Hellerstein, Google Geoff Jiang, NEC Labs Jeff Kephart, IBM Research Emre Kiciman, Microsoft Research Fabio Kon, University of S?o Paulo Mike Kozuch, Intel Labs Dejan Milojicic, HP Labs Klara Nahrstedt, UIUC Priya Narasimhan, CMU Manish Parashar, Rutgers University Ioan Raicu, Illinois Inst. of Technology Omer Rana, Cardiff University Masoud Sadjadi, Florida Intl. University Richard Schlichting, AT&T Labs Hartmut Schmeck, KIT Karsten Schwan, Georgia Tech Onn Shehory, IBM Research Eno Thereska, Microsoft Research Xiaoyun Zhu, VMware ********************************************************************** -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ ================================================================= ================================================================= From kyle at ci.uchicago.edu Thu Aug 23 21:44:28 2012 From: kyle at ci.uchicago.edu (Kyle Chard) Date: Thu, 23 Aug 2012 21:44:28 -0500 Subject: [Swift-user] CFP: DataCloud 2012 - The Third International Workshop on Data Intensive Computing in the Clouds -- Co-located with SC 12 Message-ID: <071101cd81a2$61974100$24c5c300$@ci.uchicago.edu> Call for Papers: The Third International Workshop on Data Intensive Computing in the Clouds (DataCloud 2012) November 11, Salt Lake City, UT, USA (http://www.cse.buffalo.edu/faculty/tkosar/datacloud2012) Co-Located with Super Computing 2012, November 10-16, Salt Lake City, UT, USA (http://sc12.supercomputing.org/) ---------------------------------------------------------------------------- ----------------------------------------- IMPORTANT DATES Abstract Submission: August 27, 2012 Paper Submission: September 10, 2012 Notification of Acceptance: October 8, 2012 Final Paper Due: October 29, 2012 Workshop: November 11, 2012 ---------------------------------------------------------------------------- ----------------------------------------- OVERVIEW Applications and experiments in all areas of science are becoming increasingly complex and more demanding in terms of their computational and data requirements. Some applications generate data volumes reaching hundreds of terabytes and even petabytes. As scientific applications become more data intensive, the management of data resources and dataflow between the storage and compute resources is becoming the main bottleneck. Analyzing, visualizing, and disseminating these large data sets has become a major challenge and data intensive computing is now considered as the ''fourth paradigm'' in scientific discovery after theoretical, experimental, and computational science. DataCloud 2012 will provide the scientific community a dedicated forum for discussing new research, development, and deployment efforts in running data-intensive computing workloads on Cloud Computing infrastructures. The DataCloud 2012 workshop will focus on the use of cloud-based technologies to meet the new data intensive scientific challenges that are not well served by the current supercomputers, grids or compute-intensive clouds. We believe the workshop will be an excellent place to help the community define the current state, determine future goals, and present architectures and services for future clouds supporting data intensive computing. ---------------------------------------------------------------------------- ----------------------------------------- WORKSHOP SCOPE Topics of interest include, but are not limited to: * Data-intensive cloud computing applications, characteristics, challenges * Case studies of data intensive computing in the clouds * Performance evaluation of data clouds, data grids, and data centers * Energy-efficient data cloud design and management * Data placement, scheduling, and interoperability in the clouds * Accountability, QoS, and SLAs * Data privacy and protection in a public cloud environment * Distributed file systems for clouds * Data streaming and parallelization * New programming models for data-intensive cloud computing * Scalability issues in clouds * Social computing and massively social gaming * 3D Internet and implications * Future research challenges in data-intensive cloud computing ---------------------------------------------------------------------------- ----------------------------------------- ORGANIZERS * Tevfik Kosar, University at Buffalo * Ioan Raicu, Illinois Institute of Technology & Argonne National Laboratory, USA * Roger Barga, Microsoft Research STEERING COMMITTEE * Ian Foster, University of Chicago & Argonne National Laboratory, USA * Geoffrey Fox, Indiana University * James Hamilton, Amazon Web Services * Manish Parashar, Rutgers University & National Science Foundation * Dan Reed, Microsoft Research * Rich Wolski, University of California, Santa Barbara PROGRAMME COMMITTEE . Samer Al-Kiswany, University of British Columbia, Canada . Abhishek Chandra, University of Minnesota . Rong N. ChangI, BM Research . Kyle Chard, University of Chicago & Argonne National Laboratory . Terence Critchlow, Pacific Northwest National Laboratory . Murat Demirbas, University at Buffalo . Jaliya Ekanayake, Microsoft Research . Geoffrey Fox, Indiana University . Dennis Gannon, Microsoft Research . Rob Gillen, Oak Ridge National Laboratory . Maria Indrawan, Monash University, Australia . Hui Jin, Oracle . Dan Katz, National Science Foundation . Steven Ko, University at Buffalo . Reagan Moore, University of North Carolina . Judy Qui, Indiana University . Lavanya Ramakrishnan, Lawrence Berkeley National Laboratory . Kui Ren, University at Buffalo . Yogesh Simmhan, University of Southern California, USA . Borjo Sotomayor, University of Chicago . Wei Tang, Argonne National Laboratory . Bernard Traversat, Oracle . Zhifeng Yun, Louisiana State University . Ziming Zheng, Illinois Institute of Technology -------------- next part -------------- An HTML attachment was scrubbed... URL: From iraicu at cs.iit.edu Sun Aug 26 09:17:26 2012 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Sun, 26 Aug 2012 09:17:26 -0500 Subject: [Swift-user] CFP: The 5th Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS) 2012 -- co-located with SC12 Message-ID: <503A2FF6.3080203@cs.iit.edu> Call for Papers --------------------------------------------------------------------------------------- The 5th Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS) 2012 http://datasys.cs.iit.edu/events/MTAGS12/ --------------------------------------------------------------------------------------- November 12th, 2012 Salt Lake City, Utah, USA Co-located with with IEEE/ACM International Conference for High Performance Computing, Networking, Storage and Analysis (SC12) ======================================================================================= The 5th workshop on Many-Task Computing on Grids and Supercomputers (MTAGS) will provide the scientific community a dedicated forum for presenting new research, development, and deployment efforts of large-scale many-task computing (MTC) applications on large scale clusters, Grids, Supercomputers, and Cloud Computing infrastructure. MTC, the theme of the workshop encompasses loosely coupled applications, which are generally composed of many tasks (both independent and dependent tasks) to achieve some larger application goal. This workshop will cover challenges that can hamper efficiency and utilization in running applications on large-scale systems, such as local resource manager scalability and granularity, efficient utilization of raw hardware, parallel file system contention and scalability, data management, I/O management, reliability at scale, and application scalability. We welcome paper submissions on all theoretical, simulations, and systems topics related to MTC, but we give special consideration to papers addressing petascale to exascale challenges. Papers will be peer-reviewed, and accepted papers will be published in the workshop proceedings as part of the IEEE digital library (pending approval). The workshop will be co-located with the IEEE/ACM Supercomputing 2012 Conference in Salt Lake City Utah on November 12th, 2012. For more information, please see http://datasys.cs.iit.edu/events/MTAGS12/. For more information on past workshops, please see MTAGS11, MTAGS10, MTAGS09, and MTAGS08. We also ran a Special Issue on Many-Task Computing in the IEEE Transactions on Parallel and Distributed Systems (TPDS) which has appeared in June 2011; the proceedings can be found online at http://www.computer.org/portal/web/csdl/abs/trans/td/2011/06/ttd201106toc.htm. We, the workshop organizers, also published a highly relevant paper that defines Many-Task Computing which was published in MTAGS08, titled "Many-Task Computing for Grids and Supercomputers" (http://www.cs.iit.edu/~iraicu/research/publications/2008_MTAGS08_MTC.pdf); we encourage potential authors to read this paper. Topics --------------------------------------------------------------------------------------- We invite the submission of original work that is related to the topics below. The papers can be either short (4 pages) position papers, or long (8 pages) research papers. Topics of interest include (in the context of Many-Task Computing): * Compute Resource Management * Scheduling * Job execution frameworks * Local resource manager extensions * Performance evaluation of resource managers in use on large scale systems * Dynamic resource provisioning * Techniques to manage many-core resources and/or GPUs * Challenges and opportunities in running many-task workloads on HPC systems * Challenges and opportunities in running many-task workloads on Cloud Computing infrastructure * Storage architectures and implementations * Distributed file systems * Parallel file systems * Distributed meta-data management * Content distribution systems for large data * Data caching frameworks and techniques * Data management within and across data centers * Data-aware scheduling * Data-intensive computing applications * Eventual-consistency storage usage and management * Programming models and tools * Map-reduce and its generalizations * Many-task computing middleware and applications * Parallel programming frameworks * Ensemble MPI techniques and frameworks * Service-oriented science applications * Large-Scale Workflow Systems * Workflow system performance and scalability analysis * Scalability of workflow systems * Workflow infrastructure and e-Science middleware * Programming Paradigms and Models * Large-Scale Many-Task Applications * High-throughput computing (HTC) applications * Data-intensive applications * Quasi-supercomputing applications, deployments, and experiences * Performance Evaluation * Performance evaluation * Real systems * Simulations * Reliability of large systems * How MTC Addresses Challenges of Petascale and Exascale Computing * Concurrency & Programmability * I/O & Memory * Energy * Resilience * Heterogeneity Paper Submission and Publication --------------------------------------------------------------------------------------- Authors are invited to submit papers with unpublished, original work of not more than 8 pages of double column text using single spaced 10 point size on 8.5 x 11 inch pages, as per IEEE 8.5 x 11 manuscript guidelines; document templates can be found at http://www.ieee.org/conferences_events/conferences/publishing/templates.html. We are also seeking position papers of no more than 4 pages in length. The final 4/8 page papers in PDF format must be submitted online at https://cmt.research.microsoft.com/MTAGS2012/ before the deadline. Papers will be peer-reviewed, and accepted papers will be published in the workshop proceedings as part of the IEEE digital library (pending approval). Notifications of the paper decisions will be sent out by October 12th, 2012. Selected excellent work may be eligible for additional post-conference publication as journal articles or book chapters, such as the previous Special Issue on Many-Task Computing in the IEEE Transactions on Parallel and Distributed Systems (TPDS) which has appeared in June 2011. Submission implies the willingness of at least one of the authors to register and present the paper. For more information, please http://datasys.cs.iit.edu/events/MTAGS12/, or send email to mtags12-chairs at datasys.cs.iit.edu. Important Dates --------------------------------------------------------------------------------------- * Abstract submission: September 10th, 2012 (11:59PM PST) * Paper submission: September 17th, 2012 (11:59PM PST) * Acceptance notification: October 12th, 2012 * Final papers due: November 7th, 2012 Committee Members --------------------------------------------------------------------------------------- Workshop Chairs (mtags12-chairs at datasys.cs.iit.edu) * Ioan Raicu, Illinois Institute of Technology & Argonne National Laboratory * Ian Foster, University of Chicago & Argonne National Laboratory * Yong Zhao, University of Electronic Science and Technology of China Steering Committee * David Abramson, Monash University, Australia * Jack Dongara, University of Tennessee, USA * Geoffrey Fox, Indiana University, USA * Manish Parashar, Rutgers University, USA * Marc Snir, Argonne National Laboratory & University of Illinois at Urbana Champaign, USA * Xian-He Sun, Illinois Institute of Technology, USA * Weimin Zheng, Tsinghua University, China Publicity Chair (mtags12-publicity at datasys.cs.iit.edu) * Zhao Zhang, University of Chicago, USA Program Committee Chair (mtags12-pc-chair at datasys.cs.iit.edu) * Justin Wozniak, Argonne National Laboratory, USA Technical Committee * Roger Barga, Microsoft Research, USA * Mihai Budiu, Microsoft Research, USA * Kyle Chard, University of Chicago, USA * Yong Chen, Texas Tech University, USA * Evangelinos Constantinos, Massachusetts Institute of Technology, USA * John Dennis, National Center for Atmospheric Research, USA * Catalin Dumitrescu, Fermi National Labs, USA * Dennis Gannon, Microsoft Research, USA * Indranil Gupta, University of Illinois at Urbana Champaign, USA * Florin Isaila, Universidad Carlos III de Madrid, Spain * Kamil Iskra, Argonne National Laboratory, USA * Alexandru Iosup, Delft University of Technology, Netherlands * Hui Jin, Oracle Corporation, USA * Daniel S. Katz, University of Chicago & Argonne National Laboratory, USA * Carl Kesselman, University of Southern California, USA * Zhiling Lan, Illinois Institute of Technology, USA * Mike Lang, Los Alamos National Laboratory, USA * Gregor von Laszewski, Indiana University, USA * Reagan Moore, University of North Carolina, Chappel Hill, USA * Jose Moreira, IBM Research, USA * Chris Moretti, Princeton University, USA * David O'Hallaron, Carnegie Mellon University, Intel Labs, USA * Marlon Pierce, Indiana University, USA * Judy Qiu, Indiana University, USA * Lavanya Ramakrishnan, Lawrence Berkeley National Laboratory, USA * Kui Ren, SUNY Buffalo, USA * Matei Ripeanu, University of British Columbia, Canada * Karen Schuchardt, Pacific Northwest National Laboratory, USA * Wei Tang, Argonne National Laboratory, USA * Valerie Taylor, Texas A&M, USA * Douglas Thain University of Notre Dame, USA * Edward Walker, Whitworth University, USA * Matthew Woitaszek, Occipital, Inc., USA * Ken Yocum, University of California, San Diego, USA * Zhifeng Yun, Louisiana State University, USA * Zhao Zhang, University of Chicago, USA * Ziming Zheng, Illinois Institute of Technology, USA -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ ================================================================= ================================================================= From jmargolpeople at gmail.com Tue Aug 28 10:26:51 2012 From: jmargolpeople at gmail.com (Jonathan Margoliash) Date: Tue, 28 Aug 2012 10:26:51 -0500 Subject: [Swift-user] Swift hangup error Message-ID: Hello all, I recently left a program running over night, and came back in the morning to find that it had successfully executed a swift script 14 times, but on the 15th time the swift script would not return. Specifically, the terminal output after a swift script ends is normally: ----- Final status: Tue, 28 Aug 2012 07:55:46 -0500 Finished successfully:210 Swift trunk swift-r5819 cog-r3424 RunID: 20120828-0705-4v6epf52 Returned from swift to evolve.m ----- (where the last line of text is generated by my code). However, when I came back, the last line of output was: ---- Final status: Tue, 28 Aug 2012 08:59:45 -0500 Finished successfully:210 ---- and, given the time stamp on that line, it had been hanging there for an hour. Since the two lines that usually follow the Final status line (Swift trunk and Run ID) had not been printed, I think it is clear that swift did not return control to my program. Why would this be the case? It is very improbable but possible that I ran out of space on one of the computers that swift was distributing jobs to, but I am positive that I did not run out of space on the computer which was distributing the jobs, and in any event I would hope that swift would crash more nicely if that were the case. Thanks, Jonathan -------------- next part -------------- An HTML attachment was scrubbed... URL: From wilde at mcs.anl.gov Tue Aug 28 10:43:15 2012 From: wilde at mcs.anl.gov (Michael Wilde) Date: Tue, 28 Aug 2012 10:43:15 -0500 (CDT) Subject: [Swift-user] Swift hangup error In-Reply-To: Message-ID: <2107834756.15015.1346168595242.JavaMail.root@zimbra.anl.gov> I think that there is a Swift bug which causes Swift to occasionally hang in termination processing. So if you have a script that runs many Swift scripts serially, you might need to work around this for the moment by e.g. greping for the "Final status" message and killing the Swift process, from within your multiple-swift loop. The next time this happens, please do this: First locate the PID of the swift Java process (using ps -u $USER -H) Then capture a Java stack trace of that JVM (using jstack -l thatJavaPID >& jstack.out) Then send the jstack.out file to swift-support. I can't find this in bugzilla so I'm (re)filing it as bug 821. Thanks! - Mike ----- Original Message ----- > From: "Jonathan Margoliash" > To: "Michael Wilde" , swift-user at ci.uchicago.edu, "Swift Language" , > "Professor E. Yan" > Sent: Tuesday, August 28, 2012 10:26:51 AM > Subject: Swift hangup error > Hello all, > > > I recently left a program running over night, and came back in the > morning to find that it had successfully executed a swift script 14 > times, but on the 15th time the swift script would not return. > Specifically, the terminal output after a swift script ends is > normally: > > ----- > Final status: Tue, 28 Aug 2012 07:55:46 -0500 Finished > successfully:210 > Swift trunk swift-r5819 cog-r3424 > > RunID: 20120828-0705-4v6epf52 > > Returned from swift to evolve.m > ----- > > (where the last line of text is generated by my code). However, when I > came back, the last line of output was: > > ---- > Final status: Tue, 28 Aug 2012 08:59:45 -0500 Finished > successfully:210 > > ---- > > and, given the time stamp on that line, it had been hanging there for > an hour. Since the two lines that usually follow the Final status line > (Swift trunk and Run ID) had not been printed, I think it is clear > that swift did not return control to my program. Why would this be the > case? It is very improbable but possible that I ran out of space on > one of the computers that swift was distributing jobs to, but I am > positive that I did not run out of space on the computer which was > distributing the jobs, and in any event I would hope that swift would > crash more nicely if that were the case. Thanks, > > Jonathan -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory From wilde at mcs.anl.gov Tue Aug 28 15:04:24 2012 From: wilde at mcs.anl.gov (Michael Wilde) Date: Tue, 28 Aug 2012 15:04:24 -0500 (CDT) Subject: [Swift-user] Concatenating members of a string array Message-ID: <1313924086.15693.1346184264019.JavaMail.root@zimbra.anl.gov> Sheri, in response to your off-list question: here's two ways to concatenate a string array into a single string. We should provide a primitive to do this better / more reliably. The code below is a quick hack and needs polishing for the User Guide. - Mike string sa[] = ["aaa","bbb","ccc","ddd","eee"]; string cs[]; cs[-1] = ""; foreach e,i in sa { cs[i] = @strcat(cs[i-1], " ", e); } trace("cat=",cs[@length(sa)-1]); type file; app (file o) echo (string s[]) { echo s stdout=@filename(o); } file f = echo(sa); string os = readData(f); trace("os=",os); From cphillips at mcs.anl.gov Fri Aug 31 11:19:19 2012 From: cphillips at mcs.anl.gov (Carolyn Phillips) Date: Fri, 31 Aug 2012 11:19:19 -0500 Subject: [Swift-user] using writeData Message-ID: <9DBF839C-E202-4F52-85AA-77F8A7C658A3@mcs.anl.gov> I have this loop in my script. However, when I check passes.log after the script has successfully run, it is empty! Why is that? iterate passes { # Other random stuff file passlog <"passes.log">; passlog = writeData(passes); } until (passes == 2); # Determine if Done From cphillips at mcs.anl.gov Fri Aug 31 11:42:56 2012 From: cphillips at mcs.anl.gov (Carolyn Phillips) Date: Fri, 31 Aug 2012 11:42:56 -0500 Subject: [Swift-user] using writeData In-Reply-To: <9DBF839C-E202-4F52-85AA-77F8A7C658A3@mcs.anl.gov> References: <9DBF839C-E202-4F52-85AA-77F8A7C658A3@mcs.anl.gov> Message-ID: <77F83681-FDC7-4998-8DE5-B6D0F7484CFB@mcs.anl.gov> Nm! Because the file does not end in an eol I did not notice that it was not empty at all when I cat it to the screen On Aug 31, 2012, at 11:19 AM, Carolyn Phillips wrote: > > I have this loop in my script. However, when I check passes.log after the script has successfully run, it is empty! Why is that? > > iterate passes { > > # Other random stuff > > file passlog <"passes.log">; > passlog = writeData(passes); > > } until (passes == 2); # Determine if Done From cphillips at mcs.anl.gov Fri Aug 31 12:02:21 2012 From: cphillips at mcs.anl.gov (Carolyn Phillips) Date: Fri, 31 Aug 2012 12:02:21 -0500 Subject: [Swift-user] Externally controlled loop Message-ID: <7F84B4BC-3598-4F35-A932-59328C61BD2A@mcs.anl.gov> Hello, So I want to create an externally controlled loop, meaning, the number of times the loop repeats is determined by the output of a executable called within the loop. That output is written to a file so it can be read and checked. I am experimenting with getting a very rudimentary set up. Below you can see that I have two lines that determine if the iteration is done, one based on the variable "passes" and the other based on the variable "read passes" that is read from a file. The second does not work and throws a execution failed exception. Any advice on how to make this kind of structure work? type file; iterate passes { # try writing the iteration to a log file file passlog <"passes.log">; passlog = writeData(passes); # try reading from another log file int value = readData("errorlog"); int readpasses = readData(passlog); # Write to the Output Log int i = 3; tracef("%s: %i\n", "the value is", value); tracef("%s: %i\n", "Iteration :", passes); tracef("%s: %i\n", "Iteration Read :", readpasses); #} until (readpasses == 2); # Determine if Done } until (passes == 2); # Determine if Done From hategan at mcs.anl.gov Fri Aug 31 14:17:06 2012 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Fri, 31 Aug 2012 12:17:06 -0700 Subject: [Swift-user] Externally controlled loop In-Reply-To: <7F84B4BC-3598-4F35-A932-59328C61BD2A@mcs.anl.gov> References: <7F84B4BC-3598-4F35-A932-59328C61BD2A@mcs.anl.gov> Message-ID: <1346440626.7791.1.camel@blabla> Should work. What's the exact error you are getting? $ swift b4.swift no sites file specified, setting to default: /home/mike/work/cog-svn/cogkit/trunk/current/src/cog/modules/swift/dist/swift-svn/etc/sites.xml Swift trunk swift-r5739 (swift modified locally) cog-r3088 (cog modified locally) RunID: 20120831-1215-i0n6v30e Progress: time: Fri, 31 Aug 2012 12:15:09 -0700 Iteration :: 0 Iteration Read :: 0 the value is: 1 Iteration :: 1 the value is: 1 Iteration Read :: 1 Iteration :: 2 the value is: 1 Iteration Read :: 2 Final status: Fri, 31 Aug 2012 12:15:09 -0700 $ cat b4.swift type file; iterate passes { file passlog <"passes.log">; passlog = writeData(passes); int value = readData("errorlog"); int readpasses = readData(passlog); tracef("%s: %i\n", "the value is", value); tracef("%s: %i\n", "Iteration :", passes); tracef("%s: %i\n", "Iteration Read :", readpasses); } until (readpasses == 2); # Determine if Done $ On Fri, 2012-08-31 at 12:02 -0500, Carolyn Phillips wrote: > Hello, > > So I want to create an externally controlled loop, meaning, the number of times the loop repeats is determined by the output of a executable called within the loop. That output is written to a file so it can be read and checked. I am experimenting with getting a very rudimentary set up. Below you can see that I have two lines that determine if the iteration is done, one based on the variable "passes" and the other based on the variable "read passes" that is read from a file. The second does not work and throws a execution failed exception. Any advice on how to make this kind of structure work? > > type file; > > iterate passes { > > > # try writing the iteration to a log file > file passlog <"passes.log">; > passlog = writeData(passes); > > # try reading from another log file > int value = readData("errorlog"); > int readpasses = readData(passlog); > > # Write to the Output Log > int i = 3; > tracef("%s: %i\n", "the value is", value); > tracef("%s: %i\n", "Iteration :", passes); > tracef("%s: %i\n", "Iteration Read :", readpasses); > > #} until (readpasses == 2); # Determine if Done > } until (passes == 2); # Determine if Done > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user From cphillips at mcs.anl.gov Fri Aug 31 14:20:18 2012 From: cphillips at mcs.anl.gov (Carolyn Phillips) Date: Fri, 31 Aug 2012 14:20:18 -0500 Subject: [Swift-user] Externally controlled loop In-Reply-To: <1346440626.7791.1.camel@blabla> References: <7F84B4BC-3598-4F35-A932-59328C61BD2A@mcs.anl.gov> <1346440626.7791.1.camel@blabla> Message-ID: Execution failed: java.lang.NullPointerException at org.griphyn.vdl.mapping.AbstractDataNode.getType(AbstractDataNode.java:95) at org.griphyn.vdl.karajan.lib.SwiftArg.unwrap(SwiftArg.java:57) at org.griphyn.vdl.karajan.lib.SwiftArg.getValue(SwiftArg.java:77) at org.griphyn.vdl.karajan.lib.Operators.vdlop_eq(Operators.java:216) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.globus.cog.karajan.workflow.nodes.functions.FunctionsCollection.function(FunctionsCollection.java:82) at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:27) at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194) at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28) at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29) at org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20) at org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63) at org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139) at org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197) at org.globus.cog.karajan.workflow.FlowElementWrapper.start(FlowElementWrapper.java:227) at org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:98) at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) On Aug 31, 2012, at 2:17 PM, Mihael Hategan wrote: > Should work. What's the exact error you are getting? > > $ swift b4.swift > no sites file specified, setting to > default: /home/mike/work/cog-svn/cogkit/trunk/current/src/cog/modules/swift/dist/swift-svn/etc/sites.xml > Swift trunk swift-r5739 (swift modified locally) cog-r3088 (cog modified > locally) > > RunID: 20120831-1215-i0n6v30e > Progress: time: Fri, 31 Aug 2012 12:15:09 -0700 > Iteration :: 0 > Iteration Read :: 0 > the value is: 1 > Iteration :: 1 > the value is: 1 > Iteration Read :: 1 > Iteration :: 2 > the value is: 1 > Iteration Read :: 2 > Final status: Fri, 31 Aug 2012 12:15:09 -0700 > > $ cat b4.swift > type file; > > iterate passes { > file passlog <"passes.log">; > passlog = writeData(passes); > > int value = readData("errorlog"); > int readpasses = readData(passlog); > > tracef("%s: %i\n", "the value is", value); > tracef("%s: %i\n", "Iteration :", passes); > tracef("%s: %i\n", "Iteration Read :", readpasses); > > } until (readpasses == 2); # Determine if Done > > $ > > > On Fri, 2012-08-31 at 12:02 -0500, Carolyn Phillips wrote: >> Hello, >> >> So I want to create an externally controlled loop, meaning, the number of times the loop repeats is determined by the output of a executable called within the loop. That output is written to a file so it can be read and checked. I am experimenting with getting a very rudimentary set up. Below you can see that I have two lines that determine if the iteration is done, one based on the variable "passes" and the other based on the variable "read passes" that is read from a file. The second does not work and throws a execution failed exception. Any advice on how to make this kind of structure work? >> >> type file; >> >> iterate passes { >> >> >> # try writing the iteration to a log file >> file passlog <"passes.log">; >> passlog = writeData(passes); >> >> # try reading from another log file >> int value = readData("errorlog"); >> int readpasses = readData(passlog); >> >> # Write to the Output Log >> int i = 3; >> tracef("%s: %i\n", "the value is", value); >> tracef("%s: %i\n", "Iteration :", passes); >> tracef("%s: %i\n", "Iteration Read :", readpasses); >> >> #} until (readpasses == 2); # Determine if Done >> } until (passes == 2); # Determine if Done >> _______________________________________________ >> Swift-user mailing list >> Swift-user at ci.uchicago.edu >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > From cphillips at mcs.anl.gov Fri Aug 31 14:24:03 2012 From: cphillips at mcs.anl.gov (Carolyn Phillips) Date: Fri, 31 Aug 2012 14:24:03 -0500 Subject: [Swift-user] Externally controlled loop In-Reply-To: References: <7F84B4BC-3598-4F35-A932-59328C61BD2A@mcs.anl.gov> <1346440626.7791.1.camel@blabla> Message-ID: <97AB0604-C0FB-4496-AECB-0F376B874C42@mcs.anl.gov> Just to be certain I ran your exact script cat b4.swift type file; iterate passes { file passlog <"passes.log">; passlog = writeData(passes); int value = readData("errorlog"); int readpasses = readData(passlog); tracef("%s: %i\n", "the value is", value); tracef("%s: %i\n", "Iteration :", passes); tracef("%s: %i\n", "Iteration Read :", readpasses); } until (readpasses == 2); # Determine if Done [phillicl at midway-login1 TestSwift]$ swift b4.swift no sites file specified, setting to default: /software/swift-trunk-all/etc/sites.xml Swift trunk swift-r5901 cog-r3451 RunID: 20120831-1923-58gjhdrd Progress: time: Fri, 31 Aug 2012 19:23:29 +0000 Iteration :: 0 the value is: 0 Iteration Read :: 0 Execution failed: java.lang.NullPointerException at org.griphyn.vdl.mapping.AbstractDataNode.getType(AbstractDataNode.java:95) at org.griphyn.vdl.karajan.lib.SwiftArg.unwrap(SwiftArg.java:57) at org.griphyn.vdl.karajan.lib.SwiftArg.getValue(SwiftArg.java:77) at org.griphyn.vdl.karajan.lib.Operators.vdlop_eq(Operators.java:216) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.globus.cog.karajan.workflow.nodes.functions.FunctionsCollection.function(FunctionsCollection.java:82) at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:27) at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194) at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28) at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29) at org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20) at org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63) at org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139) at org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197) at org.globus.cog.karajan.workflow.FlowElementWrapper.start(FlowElementWrapper.java:227) at org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:98) at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) On Aug 31, 2012, at 2:20 PM, Carolyn Phillips wrote: > Execution failed: > java.lang.NullPointerException > at org.griphyn.vdl.mapping.AbstractDataNode.getType(AbstractDataNode.java:95) > at org.griphyn.vdl.karajan.lib.SwiftArg.unwrap(SwiftArg.java:57) > at org.griphyn.vdl.karajan.lib.SwiftArg.getValue(SwiftArg.java:77) > at org.griphyn.vdl.karajan.lib.Operators.vdlop_eq(Operators.java:216) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.globus.cog.karajan.workflow.nodes.functions.FunctionsCollection.function(FunctionsCollection.java:82) > at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:27) > at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194) > at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) > at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) > at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28) > at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29) > at org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20) > at org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63) > at org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139) > at org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197) > at org.globus.cog.karajan.workflow.FlowElementWrapper.start(FlowElementWrapper.java:227) > at org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:98) > at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > > On Aug 31, 2012, at 2:17 PM, Mihael Hategan wrote: > >> Should work. What's the exact error you are getting? >> >> $ swift b4.swift >> no sites file specified, setting to >> default: /home/mike/work/cog-svn/cogkit/trunk/current/src/cog/modules/swift/dist/swift-svn/etc/sites.xml >> Swift trunk swift-r5739 (swift modified locally) cog-r3088 (cog modified >> locally) >> >> RunID: 20120831-1215-i0n6v30e >> Progress: time: Fri, 31 Aug 2012 12:15:09 -0700 >> Iteration :: 0 >> Iteration Read :: 0 >> the value is: 1 >> Iteration :: 1 >> the value is: 1 >> Iteration Read :: 1 >> Iteration :: 2 >> the value is: 1 >> Iteration Read :: 2 >> Final status: Fri, 31 Aug 2012 12:15:09 -0700 >> >> $ cat b4.swift >> type file; >> >> iterate passes { >> file passlog <"passes.log">; >> passlog = writeData(passes); >> >> int value = readData("errorlog"); >> int readpasses = readData(passlog); >> >> tracef("%s: %i\n", "the value is", value); >> tracef("%s: %i\n", "Iteration :", passes); >> tracef("%s: %i\n", "Iteration Read :", readpasses); >> >> } until (readpasses == 2); # Determine if Done >> >> $ >> >> >> On Fri, 2012-08-31 at 12:02 -0500, Carolyn Phillips wrote: >>> Hello, >>> >>> So I want to create an externally controlled loop, meaning, the number of times the loop repeats is determined by the output of a executable called within the loop. That output is written to a file so it can be read and checked. I am experimenting with getting a very rudimentary set up. Below you can see that I have two lines that determine if the iteration is done, one based on the variable "passes" and the other based on the variable "read passes" that is read from a file. The second does not work and throws a execution failed exception. Any advice on how to make this kind of structure work? >>> >>> type file; >>> >>> iterate passes { >>> >>> >>> # try writing the iteration to a log file >>> file passlog <"passes.log">; >>> passlog = writeData(passes); >>> >>> # try reading from another log file >>> int value = readData("errorlog"); >>> int readpasses = readData(passlog); >>> >>> # Write to the Output Log >>> int i = 3; >>> tracef("%s: %i\n", "the value is", value); >>> tracef("%s: %i\n", "Iteration :", passes); >>> tracef("%s: %i\n", "Iteration Read :", readpasses); >>> >>> #} until (readpasses == 2); # Determine if Done >>> } until (passes == 2); # Determine if Done >>> _______________________________________________ >>> Swift-user mailing list >>> Swift-user at ci.uchicago.edu >>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user >> >> > From hategan at mcs.anl.gov Fri Aug 31 14:35:12 2012 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Fri, 31 Aug 2012 12:35:12 -0700 Subject: [Swift-user] Externally controlled loop In-Reply-To: <97AB0604-C0FB-4496-AECB-0F376B874C42@mcs.anl.gov> References: <7F84B4BC-3598-4F35-A932-59328C61BD2A@mcs.anl.gov> <1346440626.7791.1.camel@blabla> <97AB0604-C0FB-4496-AECB-0F376B874C42@mcs.anl.gov> Message-ID: <1346441712.8531.0.camel@blabla> That looks like a bug in swift. What version are you running this with (I see trunk, but how did you get it)? Mihael On Fri, 2012-08-31 at 14:24 -0500, Carolyn Phillips wrote: > Just to be certain I ran your exact script > > cat b4.swift > type file; > > iterate passes { > file passlog <"passes.log">; > passlog = writeData(passes); > > int value = readData("errorlog"); > int readpasses = readData(passlog); > > tracef("%s: %i\n", "the value is", value); > tracef("%s: %i\n", "Iteration :", passes); > tracef("%s: %i\n", "Iteration Read :", readpasses); > > } until (readpasses == 2); # Determine if Done > > > [phillicl at midway-login1 TestSwift]$ swift b4.swift > no sites file specified, setting to default: /software/swift-trunk-all/etc/sites.xml > Swift trunk swift-r5901 cog-r3451 > > RunID: 20120831-1923-58gjhdrd > Progress: time: Fri, 31 Aug 2012 19:23:29 +0000 > Iteration :: 0 > the value is: 0 > Iteration Read :: 0 > Execution failed: > java.lang.NullPointerException > at org.griphyn.vdl.mapping.AbstractDataNode.getType(AbstractDataNode.java:95) > at org.griphyn.vdl.karajan.lib.SwiftArg.unwrap(SwiftArg.java:57) > at org.griphyn.vdl.karajan.lib.SwiftArg.getValue(SwiftArg.java:77) > at org.griphyn.vdl.karajan.lib.Operators.vdlop_eq(Operators.java:216) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.globus.cog.karajan.workflow.nodes.functions.FunctionsCollection.function(FunctionsCollection.java:82) > at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:27) > at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194) > at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) > at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) > at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28) > at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29) > at org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20) > at org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63) > at org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139) > at org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197) > at org.globus.cog.karajan.workflow.FlowElementWrapper.start(FlowElementWrapper.java:227) > at org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:98) > at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > > > On Aug 31, 2012, at 2:20 PM, Carolyn Phillips wrote: > > > Execution failed: > > java.lang.NullPointerException > > at org.griphyn.vdl.mapping.AbstractDataNode.getType(AbstractDataNode.java:95) > > at org.griphyn.vdl.karajan.lib.SwiftArg.unwrap(SwiftArg.java:57) > > at org.griphyn.vdl.karajan.lib.SwiftArg.getValue(SwiftArg.java:77) > > at org.griphyn.vdl.karajan.lib.Operators.vdlop_eq(Operators.java:216) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:601) > > at org.globus.cog.karajan.workflow.nodes.functions.FunctionsCollection.function(FunctionsCollection.java:82) > > at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:27) > > at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194) > > at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) > > at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) > > at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28) > > at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29) > > at org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20) > > at org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63) > > at org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139) > > at org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197) > > at org.globus.cog.karajan.workflow.FlowElementWrapper.start(FlowElementWrapper.java:227) > > at org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:98) > > at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > > at java.lang.Thread.run(Thread.java:722) > > > > On Aug 31, 2012, at 2:17 PM, Mihael Hategan wrote: > > > >> Should work. What's the exact error you are getting? > >> > >> $ swift b4.swift > >> no sites file specified, setting to > >> default: /home/mike/work/cog-svn/cogkit/trunk/current/src/cog/modules/swift/dist/swift-svn/etc/sites.xml > >> Swift trunk swift-r5739 (swift modified locally) cog-r3088 (cog modified > >> locally) > >> > >> RunID: 20120831-1215-i0n6v30e > >> Progress: time: Fri, 31 Aug 2012 12:15:09 -0700 > >> Iteration :: 0 > >> Iteration Read :: 0 > >> the value is: 1 > >> Iteration :: 1 > >> the value is: 1 > >> Iteration Read :: 1 > >> Iteration :: 2 > >> the value is: 1 > >> Iteration Read :: 2 > >> Final status: Fri, 31 Aug 2012 12:15:09 -0700 > >> > >> $ cat b4.swift > >> type file; > >> > >> iterate passes { > >> file passlog <"passes.log">; > >> passlog = writeData(passes); > >> > >> int value = readData("errorlog"); > >> int readpasses = readData(passlog); > >> > >> tracef("%s: %i\n", "the value is", value); > >> tracef("%s: %i\n", "Iteration :", passes); > >> tracef("%s: %i\n", "Iteration Read :", readpasses); > >> > >> } until (readpasses == 2); # Determine if Done > >> > >> $ > >> > >> > >> On Fri, 2012-08-31 at 12:02 -0500, Carolyn Phillips wrote: > >>> Hello, > >>> > >>> So I want to create an externally controlled loop, meaning, the number of times the loop repeats is determined by the output of a executable called within the loop. That output is written to a file so it can be read and checked. I am experimenting with getting a very rudimentary set up. Below you can see that I have two lines that determine if the iteration is done, one based on the variable "passes" and the other based on the variable "read passes" that is read from a file. The second does not work and throws a execution failed exception. Any advice on how to make this kind of structure work? > >>> > >>> type file; > >>> > >>> iterate passes { > >>> > >>> > >>> # try writing the iteration to a log file > >>> file passlog <"passes.log">; > >>> passlog = writeData(passes); > >>> > >>> # try reading from another log file > >>> int value = readData("errorlog"); > >>> int readpasses = readData(passlog); > >>> > >>> # Write to the Output Log > >>> int i = 3; > >>> tracef("%s: %i\n", "the value is", value); > >>> tracef("%s: %i\n", "Iteration :", passes); > >>> tracef("%s: %i\n", "Iteration Read :", readpasses); > >>> > >>> #} until (readpasses == 2); # Determine if Done > >>> } until (passes == 2); # Determine if Done > >>> _______________________________________________ > >>> Swift-user mailing list > >>> Swift-user at ci.uchicago.edu > >>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > >> > >> > > > From davidk at ci.uchicago.edu Fri Aug 31 14:52:34 2012 From: davidk at ci.uchicago.edu (David Kelly) Date: Fri, 31 Aug 2012 14:52:34 -0500 (CDT) Subject: [Swift-user] Externally controlled loop In-Reply-To: <1346441712.8531.0.camel@blabla> Message-ID: <134021942.54373.1346442754288.JavaMail.root@zimbra-mb2.anl.gov> This is what I get when trying with the latest trunk from SVN. I created a file called "errorlog" with a value of 1. no sites file specified, setting to default: /home/davidk/swift-trunk/cog/modules/swift/dist/swift-svn/etc/sites.xml Swift trunk swift-r5913 cog-r3459 RunID: 20120831-1448-q9y052k7 Progress: time: Fri, 31 Aug 2012 14:48:28 -0500 Iteration :: 0 the value is: 1 Iteration Read :: 0 Execution failed: java.lang.NullPointerException at org.griphyn.vdl.mapping.AbstractDataNode.getType(AbstractDataNode.java:95) at org.griphyn.vdl.karajan.lib.SwiftArg.unwrap(SwiftArg.java:57) at org.griphyn.vdl.karajan.lib.SwiftArg.getValue(SwiftArg.java:77) at org.griphyn.vdl.karajan.lib.Operators.vdlop_eq(Operators.java:216) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.globus.cog.karajan.workflow.nodes.functions.FunctionsCollection.function(FunctionsCollection.java:82) at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:27) at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194) at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28) at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29) at org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20) at org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63) at org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139) at org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197) at org.globus.cog.karajan.workflow.FlowElementWrapper.start(FlowElementWrapper.java:227) at org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:98) at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) ----- Original Message ----- > From: "Mihael Hategan" > To: "Carolyn Phillips" > Cc: swift-user at ci.uchicago.edu > Sent: Friday, August 31, 2012 2:35:12 PM > Subject: Re: [Swift-user] Externally controlled loop > That looks like a bug in swift. > > What version are you running this with (I see trunk, but how did you > get > it)? > > Mihael > > On Fri, 2012-08-31 at 14:24 -0500, Carolyn Phillips wrote: > > Just to be certain I ran your exact script > > > > cat b4.swift > > type file; > > > > iterate passes { > > file passlog <"passes.log">; > > passlog = writeData(passes); > > > > int value = readData("errorlog"); > > int readpasses = readData(passlog); > > > > tracef("%s: %i\n", "the value is", value); > > tracef("%s: %i\n", "Iteration :", passes); > > tracef("%s: %i\n", "Iteration Read :", readpasses); > > > > } until (readpasses == 2); # Determine if Done > > > > > > [phillicl at midway-login1 TestSwift]$ swift b4.swift > > no sites file specified, setting to default: > > /software/swift-trunk-all/etc/sites.xml > > Swift trunk swift-r5901 cog-r3451 > > > > RunID: 20120831-1923-58gjhdrd > > Progress: time: Fri, 31 Aug 2012 19:23:29 +0000 > > Iteration :: 0 > > the value is: 0 > > Iteration Read :: 0 > > Execution failed: > > java.lang.NullPointerException > > at > > org.griphyn.vdl.mapping.AbstractDataNode.getType(AbstractDataNode.java:95) > > at org.griphyn.vdl.karajan.lib.SwiftArg.unwrap(SwiftArg.java:57) > > at org.griphyn.vdl.karajan.lib.SwiftArg.getValue(SwiftArg.java:77) > > at > > org.griphyn.vdl.karajan.lib.Operators.vdlop_eq(Operators.java:216) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:601) > > at > > org.globus.cog.karajan.workflow.nodes.functions.FunctionsCollection.function(FunctionsCollection.java:82) > > at > > org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:27) > > at > > org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194) > > at > > org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) > > at > > org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) > > at > > org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28) > > at > > org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29) > > at > > org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20) > > at > > org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63) > > at > > org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139) > > at > > org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197) > > at > > org.globus.cog.karajan.workflow.FlowElementWrapper.start(FlowElementWrapper.java:227) > > at > > org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:98) > > at > > org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > > at > > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > > at java.lang.Thread.run(Thread.java:722) > > > > > > On Aug 31, 2012, at 2:20 PM, Carolyn Phillips > > wrote: > > > > > Execution failed: > > > java.lang.NullPointerException > > > at > > > org.griphyn.vdl.mapping.AbstractDataNode.getType(AbstractDataNode.java:95) > > > at org.griphyn.vdl.karajan.lib.SwiftArg.unwrap(SwiftArg.java:57) > > > at > > > org.griphyn.vdl.karajan.lib.SwiftArg.getValue(SwiftArg.java:77) > > > at > > > org.griphyn.vdl.karajan.lib.Operators.vdlop_eq(Operators.java:216) > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > at > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > at java.lang.reflect.Method.invoke(Method.java:601) > > > at > > > org.globus.cog.karajan.workflow.nodes.functions.FunctionsCollection.function(FunctionsCollection.java:82) > > > at > > > org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:27) > > > at > > > org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194) > > > at > > > org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) > > > at > > > org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) > > > at > > > org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28) > > > at > > > org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29) > > > at > > > org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20) > > > at > > > org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63) > > > at > > > org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139) > > > at > > > org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197) > > > at > > > org.globus.cog.karajan.workflow.FlowElementWrapper.start(FlowElementWrapper.java:227) > > > at > > > org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:98) > > > at > > > org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) > > > at > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > > > at > > > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > > > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > > > at java.lang.Thread.run(Thread.java:722) > > > > > > On Aug 31, 2012, at 2:17 PM, Mihael Hategan > > > wrote: > > > > > >> Should work. What's the exact error you are getting? > > >> > > >> $ swift b4.swift > > >> no sites file specified, setting to > > >> default: > > >> /home/mike/work/cog-svn/cogkit/trunk/current/src/cog/modules/swift/dist/swift-svn/etc/sites.xml > > >> Swift trunk swift-r5739 (swift modified locally) cog-r3088 (cog > > >> modified > > >> locally) > > >> > > >> RunID: 20120831-1215-i0n6v30e > > >> Progress: time: Fri, 31 Aug 2012 12:15:09 -0700 > > >> Iteration :: 0 > > >> Iteration Read :: 0 > > >> the value is: 1 > > >> Iteration :: 1 > > >> the value is: 1 > > >> Iteration Read :: 1 > > >> Iteration :: 2 > > >> the value is: 1 > > >> Iteration Read :: 2 > > >> Final status: Fri, 31 Aug 2012 12:15:09 -0700 > > >> > > >> $ cat b4.swift > > >> type file; > > >> > > >> iterate passes { > > >> file passlog <"passes.log">; > > >> passlog = writeData(passes); > > >> > > >> int value = readData("errorlog"); > > >> int readpasses = readData(passlog); > > >> > > >> tracef("%s: %i\n", "the value is", value); > > >> tracef("%s: %i\n", "Iteration :", passes); > > >> tracef("%s: %i\n", "Iteration Read :", readpasses); > > >> > > >> } until (readpasses == 2); # Determine if Done > > >> > > >> $ > > >> > > >> > > >> On Fri, 2012-08-31 at 12:02 -0500, Carolyn Phillips wrote: > > >>> Hello, > > >>> > > >>> So I want to create an externally controlled loop, meaning, the > > >>> number of times the loop repeats is determined by the output of > > >>> a executable called within the loop. That output is written to a > > >>> file so it can be read and checked. I am experimenting with > > >>> getting a very rudimentary set up. Below you can see that I have > > >>> two lines that determine if the iteration is done, one based on > > >>> the variable "passes" and the other based on the variable "read > > >>> passes" that is read from a file. The second does not work and > > >>> throws a execution failed exception. Any advice on how to make > > >>> this kind of structure work? > > >>> > > >>> type file; > > >>> > > >>> iterate passes { > > >>> > > >>> > > >>> # try writing the iteration to a log file > > >>> file passlog <"passes.log">; > > >>> passlog = writeData(passes); > > >>> > > >>> # try reading from another log file > > >>> int value = readData("errorlog"); > > >>> int readpasses = readData(passlog); > > >>> > > >>> # Write to the Output Log > > >>> int i = 3; > > >>> tracef("%s: %i\n", "the value is", value); > > >>> tracef("%s: %i\n", "Iteration :", passes); > > >>> tracef("%s: %i\n", "Iteration Read :", readpasses); > > >>> > > >>> #} until (readpasses == 2); # Determine if Done > > >>> } until (passes == 2); # Determine if Done > > >>> _______________________________________________ > > >>> Swift-user mailing list > > >>> Swift-user at ci.uchicago.edu > > >>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > >> > > >> > > > > > > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user From cphillips at mcs.anl.gov Fri Aug 31 14:53:14 2012 From: cphillips at mcs.anl.gov (Carolyn Phillips) Date: Fri, 31 Aug 2012 14:53:14 -0500 Subject: [Swift-user] Externally controlled loop In-Reply-To: <1346441712.8531.0.camel@blabla> References: <7F84B4BC-3598-4F35-A932-59328C61BD2A@mcs.anl.gov> <1346440626.7791.1.camel@blabla> <97AB0604-C0FB-4496-AECB-0F376B874C42@mcs.anl.gov> <1346441712.8531.0.camel@blabla> Message-ID: I am working on Miday run by RCC at UChicago [phillicl at midway-login2 ~]$ module display swift ------------------------------------------------------------------- /software/modulefiles/swift/trunk: module-whatis setup swift trunk compiled with the system compiler conflict swift module load java/1.7 prepend-path PATH /software/swift-trunk-all/bin prepend-path LD_LIBRARY_PATH /software/swift-trunk-all/lib prepend-path LIBRARY_PATH /software/swift-trunk-all/lib ------------------------------------------------------------------- [phillicl at midway-login2 ~]$ I On Aug 31, 2012, at 2:35 PM, Mihael Hategan wrote: > That looks like a bug in swift. > > What version are you running this with (I see trunk, but how did you get > it)? > > Mihael > > On Fri, 2012-08-31 at 14:24 -0500, Carolyn Phillips wrote: >> Just to be certain I ran your exact script >> >> cat b4.swift >> type file; >> >> iterate passes { >> file passlog <"passes.log">; >> passlog = writeData(passes); >> >> int value = readData("errorlog"); >> int readpasses = readData(passlog); >> >> tracef("%s: %i\n", "the value is", value); >> tracef("%s: %i\n", "Iteration :", passes); >> tracef("%s: %i\n", "Iteration Read :", readpasses); >> >> } until (readpasses == 2); # Determine if Done >> >> >> [phillicl at midway-login1 TestSwift]$ swift b4.swift >> no sites file specified, setting to default: /software/swift-trunk-all/etc/sites.xml >> Swift trunk swift-r5901 cog-r3451 >> >> RunID: 20120831-1923-58gjhdrd >> Progress: time: Fri, 31 Aug 2012 19:23:29 +0000 >> Iteration :: 0 >> the value is: 0 >> Iteration Read :: 0 >> Execution failed: >> java.lang.NullPointerException >> at org.griphyn.vdl.mapping.AbstractDataNode.getType(AbstractDataNode.java:95) >> at org.griphyn.vdl.karajan.lib.SwiftArg.unwrap(SwiftArg.java:57) >> at org.griphyn.vdl.karajan.lib.SwiftArg.getValue(SwiftArg.java:77) >> at org.griphyn.vdl.karajan.lib.Operators.vdlop_eq(Operators.java:216) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:601) >> at org.globus.cog.karajan.workflow.nodes.functions.FunctionsCollection.function(FunctionsCollection.java:82) >> at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:27) >> at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194) >> at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) >> at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) >> at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28) >> at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29) >> at org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20) >> at org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63) >> at org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139) >> at org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197) >> at org.globus.cog.karajan.workflow.FlowElementWrapper.start(FlowElementWrapper.java:227) >> at org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:98) >> at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) >> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >> at java.lang.Thread.run(Thread.java:722) >> >> >> On Aug 31, 2012, at 2:20 PM, Carolyn Phillips wrote: >> >>> Execution failed: >>> java.lang.NullPointerException >>> at org.griphyn.vdl.mapping.AbstractDataNode.getType(AbstractDataNode.java:95) >>> at org.griphyn.vdl.karajan.lib.SwiftArg.unwrap(SwiftArg.java:57) >>> at org.griphyn.vdl.karajan.lib.SwiftArg.getValue(SwiftArg.java:77) >>> at org.griphyn.vdl.karajan.lib.Operators.vdlop_eq(Operators.java:216) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:601) >>> at org.globus.cog.karajan.workflow.nodes.functions.FunctionsCollection.function(FunctionsCollection.java:82) >>> at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:27) >>> at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194) >>> at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) >>> at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) >>> at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28) >>> at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29) >>> at org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20) >>> at org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63) >>> at org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139) >>> at org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197) >>> at org.globus.cog.karajan.workflow.FlowElementWrapper.start(FlowElementWrapper.java:227) >>> at org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:98) >>> at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) >>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >>> at java.lang.Thread.run(Thread.java:722) >>> >>> On Aug 31, 2012, at 2:17 PM, Mihael Hategan wrote: >>> >>>> Should work. What's the exact error you are getting? >>>> >>>> $ swift b4.swift >>>> no sites file specified, setting to >>>> default: /home/mike/work/cog-svn/cogkit/trunk/current/src/cog/modules/swift/dist/swift-svn/etc/sites.xml >>>> Swift trunk swift-r5739 (swift modified locally) cog-r3088 (cog modified >>>> locally) >>>> >>>> RunID: 20120831-1215-i0n6v30e >>>> Progress: time: Fri, 31 Aug 2012 12:15:09 -0700 >>>> Iteration :: 0 >>>> Iteration Read :: 0 >>>> the value is: 1 >>>> Iteration :: 1 >>>> the value is: 1 >>>> Iteration Read :: 1 >>>> Iteration :: 2 >>>> the value is: 1 >>>> Iteration Read :: 2 >>>> Final status: Fri, 31 Aug 2012 12:15:09 -0700 >>>> >>>> $ cat b4.swift >>>> type file; >>>> >>>> iterate passes { >>>> file passlog <"passes.log">; >>>> passlog = writeData(passes); >>>> >>>> int value = readData("errorlog"); >>>> int readpasses = readData(passlog); >>>> >>>> tracef("%s: %i\n", "the value is", value); >>>> tracef("%s: %i\n", "Iteration :", passes); >>>> tracef("%s: %i\n", "Iteration Read :", readpasses); >>>> >>>> } until (readpasses == 2); # Determine if Done >>>> >>>> $ >>>> >>>> >>>> On Fri, 2012-08-31 at 12:02 -0500, Carolyn Phillips wrote: >>>>> Hello, >>>>> >>>>> So I want to create an externally controlled loop, meaning, the number of times the loop repeats is determined by the output of a executable called within the loop. That output is written to a file so it can be read and checked. I am experimenting with getting a very rudimentary set up. Below you can see that I have two lines that determine if the iteration is done, one based on the variable "passes" and the other based on the variable "read passes" that is read from a file. The second does not work and throws a execution failed exception. Any advice on how to make this kind of structure work? >>>>> >>>>> type file; >>>>> >>>>> iterate passes { >>>>> >>>>> >>>>> # try writing the iteration to a log file >>>>> file passlog <"passes.log">; >>>>> passlog = writeData(passes); >>>>> >>>>> # try reading from another log file >>>>> int value = readData("errorlog"); >>>>> int readpasses = readData(passlog); >>>>> >>>>> # Write to the Output Log >>>>> int i = 3; >>>>> tracef("%s: %i\n", "the value is", value); >>>>> tracef("%s: %i\n", "Iteration :", passes); >>>>> tracef("%s: %i\n", "Iteration Read :", readpasses); >>>>> >>>>> #} until (readpasses == 2); # Determine if Done >>>>> } until (passes == 2); # Determine if Done >>>>> _______________________________________________ >>>>> Swift-user mailing list >>>>> Swift-user at ci.uchicago.edu >>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user >>>> >>>> >>> >> > > From cphillips at mcs.anl.gov Fri Aug 31 18:55:17 2012 From: cphillips at mcs.anl.gov (Carolyn Phillips) Date: Fri, 31 Aug 2012 18:55:17 -0500 Subject: [Swift-user] Output Files, ReadData and Order of Execution Message-ID: <12C4B0C3-95C6-42B4-BC95-B07588359E58@mcs.anl.gov> So I execute an atomic procedure to generate a datafile, and then next I want to do something with that data file. However, my program is trying to do something with the datafile before it has been written to. So something with order of execution is not working. I think the problem is that the name of my file exists, but the file itself does not yet, but execution proceeds anyway! Here are my lines type pointfile { unlabeleddata points; errorlog error; } # Generate Parameters pointfile np ; np = generatepoints(config,labeledpoints, "uniform", 50); int checkforerror = readData(@np.error); This gives an error : mypoints.error.dat (No such file or directory) If I comment out the last line.. all the files show up in the directory. (e.g. mypoints.points.dat and mypoints.error.dat) ) and if forget to remove the .dat files from a prior run, it also runs fine! How do you fix a problem like that? From hategan at mcs.anl.gov Fri Aug 31 19:17:52 2012 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Fri, 31 Aug 2012 17:17:52 -0700 Subject: [Swift-user] Output Files, ReadData and Order of Execution In-Reply-To: <12C4B0C3-95C6-42B4-BC95-B07588359E58@mcs.anl.gov> References: <12C4B0C3-95C6-42B4-BC95-B07588359E58@mcs.anl.gov> Message-ID: <1346458672.11980.1.camel@blabla> @np.error means the file name of np.error which is known statically. So readData(@np.error) can run as soon as the script starts. You probably want to say readData(np.error). Mihael On Fri, 2012-08-31 at 18:55 -0500, Carolyn Phillips wrote: > So I execute an atomic procedure to generate a datafile, and then next > I want to do something with that data file. However, my program is > trying to do something with the datafile before it has been written > to. So something with order of execution is not working. I think the > problem is that the name of my file exists, but the file itself does > not yet, but execution proceeds anyway! > > Here are my lines > > type pointfile { > unlabeleddata points; > errorlog error; > } > > # Generate Parameters > pointfile np ; > np = generatepoints(config,labeledpoints, "uniform", 50); > > int checkforerror = readData(@np.error); > > This gives an error : > mypoints.error.dat (No such file or directory) > > If I comment out the last line.. all the files show up in the directory. (e.g. mypoints.points.dat and mypoints.error.dat) ) and if forget to remove the .dat files from a prior run, it also runs fine! > > How do you fix a problem like that? > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user From cphillips at mcs.anl.gov Fri Aug 31 19:29:50 2012 From: cphillips at mcs.anl.gov (Carolyn Phillips) Date: Fri, 31 Aug 2012 19:29:50 -0500 Subject: [Swift-user] Output Files, ReadData and Order of Execution In-Reply-To: <1346458672.11980.1.camel@blabla> References: <12C4B0C3-95C6-42B4-BC95-B07588359E58@mcs.anl.gov> <1346458672.11980.1.camel@blabla> Message-ID: Hi Mihael, the reason I added the "@" was because now this (similar) line if(checkforerror==0) { string parameters[] =readData(np.points); } gives me this: Execution failed: mypoints..dat (No such file or directory) as in now its not getting the name of the file correct On Aug 31, 2012, at 7:17 PM, Mihael Hategan wrote: > @np.error means the file name of np.error which is known statically. So > readData(@np.error) can run as soon as the script starts. > > You probably want to say readData(np.error). > > Mihael > > > On Fri, 2012-08-31 at 18:55 -0500, Carolyn Phillips wrote: >> So I execute an atomic procedure to generate a datafile, and then next >> I want to do something with that data file. However, my program is >> trying to do something with the datafile before it has been written >> to. So something with order of execution is not working. I think the >> problem is that the name of my file exists, but the file itself does >> not yet, but execution proceeds anyway! >> >> Here are my lines >> >> type pointfile { >> unlabeleddata points; >> errorlog error; >> } >> >> # Generate Parameters >> pointfile np ; >> np = generatepoints(config,labeledpoints, "uniform", 50); >> >> int checkforerror = readData(@np.error); >> >> This gives an error : >> mypoints.error.dat (No such file or directory) >> >> If I comment out the last line.. all the files show up in the directory. (e.g. mypoints.points.dat and mypoints.error.dat) ) and if forget to remove the .dat files from a prior run, it also runs fine! >> >> How do you fix a problem like that? >> >> _______________________________________________ >> Swift-user mailing list >> Swift-user at ci.uchicago.edu >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > From hategan at mcs.anl.gov Fri Aug 31 19:49:31 2012 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Fri, 31 Aug 2012 17:49:31 -0700 Subject: [Swift-user] Output Files, ReadData and Order of Execution In-Reply-To: References: <12C4B0C3-95C6-42B4-BC95-B07588359E58@mcs.anl.gov> <1346458672.11980.1.camel@blabla> Message-ID: <1346460571.12165.0.camel@blabla> Another bug. I committed a fix. In the mean time, the solution is: errorlog fe = np.errorlog; int error = readData(fe); On Fri, 2012-08-31 at 19:29 -0500, Carolyn Phillips wrote: > Hi Mihael, > > the reason I added the "@" was because > > now this (similar) line > > if(checkforerror==0) { > string parameters[] =readData(np.points); > } > > gives me this: > > Execution failed: > mypoints..dat (No such file or directory) > > as in now its not getting the name of the file correct > > On Aug 31, 2012, at 7:17 PM, Mihael Hategan wrote: > > > @np.error means the file name of np.error which is known statically. So > > readData(@np.error) can run as soon as the script starts. > > > > You probably want to say readData(np.error). > > > > Mihael > > > > > > On Fri, 2012-08-31 at 18:55 -0500, Carolyn Phillips wrote: > >> So I execute an atomic procedure to generate a datafile, and then next > >> I want to do something with that data file. However, my program is > >> trying to do something with the datafile before it has been written > >> to. So something with order of execution is not working. I think the > >> problem is that the name of my file exists, but the file itself does > >> not yet, but execution proceeds anyway! > >> > >> Here are my lines > >> > >> type pointfile { > >> unlabeleddata points; > >> errorlog error; > >> } > >> > >> # Generate Parameters > >> pointfile np ; > >> np = generatepoints(config,labeledpoints, "uniform", 50); > >> > >> int checkforerror = readData(@np.error); > >> > >> This gives an error : > >> mypoints.error.dat (No such file or directory) > >> > >> If I comment out the last line.. all the files show up in the directory. (e.g. mypoints.points.dat and mypoints.error.dat) ) and if forget to remove the .dat files from a prior run, it also runs fine! > >> > >> How do you fix a problem like that? > >> > >> _______________________________________________ > >> Swift-user mailing list > >> Swift-user at ci.uchicago.edu > >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > > > > From cphillips at mcs.anl.gov Fri Aug 31 20:11:28 2012 From: cphillips at mcs.anl.gov (Carolyn Phillips) Date: Fri, 31 Aug 2012 20:11:28 -0500 Subject: [Swift-user] Output Files, ReadData and Order of Execution In-Reply-To: <1346460571.12165.0.camel@blabla> References: <12C4B0C3-95C6-42B4-BC95-B07588359E58@mcs.anl.gov> <1346458672.11980.1.camel@blabla> <1346460571.12165.0.camel@blabla> Message-ID: <98600E77-C847-4A5F-A2CD-6B134543E630@mcs.anl.gov> How would this line work for what I have below? >> string parameters[] =readData(np.points); On Aug 31, 2012, at 7:49 PM, Mihael Hategan wrote: > Another bug. > > I committed a fix. In the mean time, the solution is: > > > errorlog fe = np.errorlog; > > int error = readData(fe); > > On Fri, 2012-08-31 at 19:29 -0500, Carolyn Phillips wrote: >> Hi Mihael, >> >> the reason I added the "@" was because >> >> now this (similar) line >> >> if(checkforerror==0) { >> string parameters[] =readData(np.points); >> } >> >> gives me this: >> >> Execution failed: >> mypoints..dat (No such file or directory) >> >> as in now its not getting the name of the file correct >> >> On Aug 31, 2012, at 7:17 PM, Mihael Hategan wrote: >> >>> @np.error means the file name of np.error which is known statically. So >>> readData(@np.error) can run as soon as the script starts. >>> >>> You probably want to say readData(np.error). >>> >>> Mihael >>> >>> >>> On Fri, 2012-08-31 at 18:55 -0500, Carolyn Phillips wrote: >>>> So I execute an atomic procedure to generate a datafile, and then next >>>> I want to do something with that data file. However, my program is >>>> trying to do something with the datafile before it has been written >>>> to. So something with order of execution is not working. I think the >>>> problem is that the name of my file exists, but the file itself does >>>> not yet, but execution proceeds anyway! >>>> >>>> Here are my lines >>>> >>>> type pointfile { >>>> unlabeleddata points; >>>> errorlog error; >>>> } >>>> >>>> # Generate Parameters >>>> pointfile np ; >>>> np = generatepoints(config,labeledpoints, "uniform", 50); >>>> >>>> int checkforerror = readData(@np.error); >>>> >>>> This gives an error : >>>> mypoints.error.dat (No such file or directory) >>>> >>>> If I comment out the last line.. all the files show up in the directory. (e.g. mypoints.points.dat and mypoints.error.dat) ) and if forget to remove the .dat files from a prior run, it also runs fine! >>>> >>>> How do you fix a problem like that? >>>> >>>> _______________________________________________ >>>> Swift-user mailing list >>>> Swift-user at ci.uchicago.edu >>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user >>> >>> >> > > From hategan at mcs.anl.gov Fri Aug 31 20:27:48 2012 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Fri, 31 Aug 2012 18:27:48 -0700 Subject: [Swift-user] Output Files, ReadData and Order of Execution In-Reply-To: <98600E77-C847-4A5F-A2CD-6B134543E630@mcs.anl.gov> References: <12C4B0C3-95C6-42B4-BC95-B07588359E58@mcs.anl.gov> <1346458672.11980.1.camel@blabla> <1346460571.12165.0.camel@blabla> <98600E77-C847-4A5F-A2CD-6B134543E630@mcs.anl.gov> Message-ID: <1346462868.17915.0.camel@blabla> On Fri, 2012-08-31 at 20:11 -0500, Carolyn Phillips wrote: > How would this line work for what I have below? > > >> string parameters[] =readData(np.points); > unlabeleddata tmp = np.points; string parameters[] = readData(tmp); > > > > On Aug 31, 2012, at 7:49 PM, Mihael Hategan wrote: > > > Another bug. > > > > I committed a fix. In the mean time, the solution is: > > > > > > errorlog fe = np.errorlog; > > > > int error = readData(fe); > > > > On Fri, 2012-08-31 at 19:29 -0500, Carolyn Phillips wrote: > >> Hi Mihael, > >> > >> the reason I added the "@" was because > >> > >> now this (similar) line > >> > >> if(checkforerror==0) { > >> string parameters[] =readData(np.points); > >> } > >> > >> gives me this: > >> > >> Execution failed: > >> mypoints..dat (No such file or directory) > >> > >> as in now its not getting the name of the file correct > >> > >> On Aug 31, 2012, at 7:17 PM, Mihael Hategan wrote: > >> > >>> @np.error means the file name of np.error which is known statically. So > >>> readData(@np.error) can run as soon as the script starts. > >>> > >>> You probably want to say readData(np.error). > >>> > >>> Mihael > >>> > >>> > >>> On Fri, 2012-08-31 at 18:55 -0500, Carolyn Phillips wrote: > >>>> So I execute an atomic procedure to generate a datafile, and then next > >>>> I want to do something with that data file. However, my program is > >>>> trying to do something with the datafile before it has been written > >>>> to. So something with order of execution is not working. I think the > >>>> problem is that the name of my file exists, but the file itself does > >>>> not yet, but execution proceeds anyway! > >>>> > >>>> Here are my lines > >>>> > >>>> type pointfile { > >>>> unlabeleddata points; > >>>> errorlog error; > >>>> } > >>>> > >>>> # Generate Parameters > >>>> pointfile np ; > >>>> np = generatepoints(config,labeledpoints, "uniform", 50); > >>>> > >>>> int checkforerror = readData(@np.error); > >>>> > >>>> This gives an error : > >>>> mypoints.error.dat (No such file or directory) > >>>> > >>>> If I comment out the last line.. all the files show up in the directory. (e.g. mypoints.points.dat and mypoints.error.dat) ) and if forget to remove the .dat files from a prior run, it also runs fine! > >>>> > >>>> How do you fix a problem like that? > >>>> > >>>> _______________________________________________ > >>>> Swift-user mailing list > >>>> Swift-user at ci.uchicago.edu > >>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > >>> > >>> > >> > > > > >