From thomas.schenker at student.kit.edu Fri Nov 2 10:18:20 2012 From: thomas.schenker at student.kit.edu (Schenker, Thomas) Date: Fri, 2 Nov 2012 16:18:20 +0100 Subject: [Swift-user] Bag of Workstatitions connection timeout Message-ID: <5FFA6092D0DC9743BC59D656F9F8D4DD0276CBB963EB@KIT-MSX-14.kit.edu> Hi, i'm trying to run swift on some workstations, following these instructions: http://www.ci.uchicago.edu/swift/guides/release-0.93/siteguide/siteguide.html#_bag_of_workstations . My coaster-service.conf looks like this: # Location of SWIFT. If empty, PATH is searched export SWIFT=/home/tschenker/swift-0.93/bin/swift # Where to copy worker.pl on the remote machine for sites.xml #export WORKER_LOCATION=$HOME export WORKER_LOCATION=/home/ubuntu # How to launch workers: local, ssh, cobalt, or futuregrid export WORKER_MODE=ssh # SSH hosts to start workers on (ssh mode only) export WORKER_HOSTS="10.1.218.228 10.1.218.229" # Do all the worker nodes you're using have a shared filesystem? (yes/no) export SHARED_FILESYSTEM=no # Username to use on worker nodes export WORKER_USERNAME=ubuntu # Enable SSH tunneling? (yes/no) export SSH_TUNNELING=no # Directory to keep log files, relative to working directory when launching start-coaster-service export LOG_DIR=logs # Manually define ports. If not specified, an available port will be used export LOCAL_PORT= export SERVICE_PORT= # This is the IP address to which the workers will connect # If not given, start-coaster-service tries to automatically detect # the IP address of this system via ifconfig # Specify this if you have multiple network interfaces export IPADDR= # Location of the swift-vm-boot scripts export SWIFTVMBOOT_DIR=$HOME/swift-vm-boot # Swift information for creating sites.xml export WORK=/home/tschenker/.work_swift export QUEUE=prod-devel export MAXTIME=20 export NODE=64 export JOBS_PER_NODE=1 export JOB_THROTTLE=0.019 start-coaster-service starts worker.pl on the workstations, but after ~30 seconds these processes die. This is what i get when i run one of the swift examples: ~$ swift -sites.file ~/sites.xml -tc.file ~/tc.data -config cf swift-0.93/examples/swift/tutorial/if.swift Swift 0.93 swift-r5483 cog-r3339 RunID: 20121029-2128-urhpc6eg Failed to acquire exclusive lock on log file. Progress: time: Mon, 29 Oct 2012 21:28:04 +0100 Find: http://10.1.167.72:57254 Find: keepalive(120), reconnect - http://10.1.167.72:57254 Passive queue processor initialized. Callback URI is http://127.0.1.1:43284 Progress: time: Mon, 29 Oct 2012 21:28:34 +0100 Submitted:1 Progress: time: Mon, 29 Oct 2012 21:29:04 +0100 Submitted:1 Failed to connect: Connection timed out at /home/ubuntu/worker.pl line 372. Failed to connect: Connection timed out at /home/ubuntu/worker.pl line 372. Progress: time: Mon, 29 Oct 2012 21:29:34 +0100 Submitted:1 Progress: time: Mon, 29 Oct 2012 21:30:04 +0100 Submitted:1 ... Can anybody help? Thanks, Thomas From iraicu at cs.iit.edu Sun Nov 4 19:47:34 2012 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Sun, 04 Nov 2012 19:47:34 -0600 Subject: [Swift-user] Call for Participation: IEEE MTAGS 2012 at SC12 on November 12th -- Win a Google Nexus 7! Message-ID: <50971AB6.1090404@cs.iit.edu> Call for Participation --------------------------------------------------------------------------------------- The 5th Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS) 2012 http://datasys.cs.iit.edu/events/MTAGS12/ --------------------------------------------------------------------------------------- November 12th, 2012 Salt Lake City, Utah, USA Co-located with with IEEE/ACM International Conference for High Performance Computing, Networking, Storage and Analysis (SC12) Location: 155-C Date: November 12th, 2012 Time: 9AM - 5:30PM ======================================================================================= The 5th workshop on Many-Task Computing on Grids and Supercomputers (MTAGS) will provide the scientific community a dedicated forum for presenting new research, development, and deployment efforts of large-scale many-task computing (MTC) applications on large scale clusters, Grids, Supercomputers, and Cloud Computing infrastructure. The workshop will be co-located with the IEEE/ACM Supercomputing 2012 Conference in Salt Lake City Utah on November 12th, 2012. For more information, please see http://datasys.cs.iit.edu/events/MTAGS12/. Some highlights of the upcoming workshop program: * Win a Google Nexus 7 tablet (visit http://datasys.cs.iit.edu/events/MTAGS12/prize.html); Must be present at 5:30PM on November 12th, 2012, at the workshop to win. * Keynote Talk: Adaptive Runtime Systems meet needs of many task computing by Dr. Laxmikant (Sanjay) Kale, Professor of Computer Science, University of Illinois at Urbana Champaign * Invited Talk: Petascale Challenge Award: Data Management for Parallel Scripting, by Zhao Zhang (UChicago) * Invited Talk: Biggest Impact Award: IaaS Cloud Benchmarking: Approaches, Challenges, and Experience, by Alex Iosup (TUDelft) * Invited Talk: Cloud Challenge Award: Portable Data Mining on Azure and HPC Platforms, by Judy Qiu (IndianaU) * Paper Talk: Accessible Datastore of High-Throughput Calculations: Experiences from the Materials Project * Paper Talk: Resource Management for Dynamic MapReduce Clusters in Multicluster Systems * Paper Talk: A Comparative Study of Data Processing Approaches for Text Processing Workflows * Paper Talk: A Scalable Master-Worker Architecture for PaaS Clouds * Paper Talk: HOG:Distributed Hadoop MapReduce on the Grid * Paper Talk: A Hybrid Scheduling Approach for Scalable Heterogeneous Hadoop Systems * Paper Talk: Software-as-a-Service: The iPlant Foundation API ======================================================================================= General Chairs Ioan Raicu, Illinois Institute of Technology & Argonne National Laboratory, USA Ian Foster, University of Chicago & Argonne National Laboratory, USA Yong Zhao, University of Electronic Science and Technology of China, China Program Committee Chair Justin Wozniak, Argonne National Laboratory, USA ======================================================================================= Generous sponsorship has been confirmed from University of Chicago (Computation Institute) and Illinois Institute of Technology (College of Science and Letters, and Graduate School)! See http://datasys.cs.iit.edu/events/MTAGS12/index.html#Sponsors for more details. -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From lpesce at uchicago.edu Tue Nov 6 11:21:48 2012 From: lpesce at uchicago.edu (Lorenzo Pesce) Date: Tue, 6 Nov 2012 11:21:48 -0600 Subject: [Swift-user] Problems with mappers on Beagle Message-ID: Hi -- We are trying to run a swift script that would run a set of java scripts. We are attempting to use swift 0.94xxxx, the version Mike installed on Beagle. I am afraid that I am not understanding how swift sees filesystems. Basically what we want to run is something like this /rootdir /rootdir/workdir /rootdir/datadir swift is run in workdir, looks for input files in datadir and builds the same directory structure in workdir. All files are Direct. I am a worried that I incorrectly set absolute/relative paths. Currently the input file is set as a string, that was for debugging purposes and we would like to leave it as a file with Direct. Location: /lustre/beagle/pittjj/SwiftRun screen shot: lpesce at login5:/lustre/beagle/pittjj/SwiftRun> cat gatk_post.swift_2012-11-06_11:14:20 Swift trunk swift-r6003 cog-r3497 RunID: 20121106-1714-4tymbt3c Progress: time: Tue, 06 Nov 2012 17:14:23 +0000 Execution failed: Illegal extra argument `Dirs[0]:string = TCGA-AR-A24T-01A-11R-A169-07.accepted_hits - Closed' to _g_a_t_kpost @ gatk_post.kml, line: 186 Thanks Lorenzo From isadoogh at iit.edu Thu Nov 8 20:34:38 2012 From: isadoogh at iit.edu (Iman Sadooghi) Date: Thu, 8 Nov 2012 20:34:38 -0600 Subject: [Swift-user] coaster service doesn't work properly. Message-ID: I'm having problem trying to run swift on multiple nodes. I 'm using coaster service. here is the contents of my coaster service: ----------------------------------------------------------------------------------------------------------------------------------------------------------- # Keep all interesting settings in one place # User should modify this to fit environment # Location of SWIFT. If empty, PATH is searched export SWIFT= # Where to copy worker.pl on the remote machine for sites.xml export WORKER_LOCATION=$HOME/swiftwork # How to launch workers: local, ssh, cobalt, or futuregrid export WORKER_MODE=ssh # SSH hosts to start workers on (ssh mode only) export WORKER_HOSTS="216.47.142.127" # Do all the worker nodes you're using have a shared filesystem? (yes/no) export SHARED_FILESYSTEM=no # Username to use on worker nodes export WORKER_USERNAME=$iman # Enable SSH tunneling? (yes/no) export SSH_TUNNELING=yes # Directory to keep log files, relative to working directory when launching start-coaster-service export LOG_DIR=$HOME/logs # Manually define ports. If not specified, an available port will be used export LOCAL_PORT= export SERVICE_PORT= # This is the IP address to which the workers will connect # If not given, start-coaster-service tries to automatically detect # the IP address of this system via ifconfig # Specify this if you have multiple network interfaces export IPADDR= # Location of the swift-vm-boot scripts export SWIFTVMBOOT_DIR=$HOME/swift-vm-boot # Swift information for creating sites.xml export WORK=/tmp export QUEUE=prod-devel export MAXTIME=20 export NODE=64 export JOBS_PER_NODE=1 export JOB_THROTTLE=0.799 ------------------------------------------------------------------------------------------------------------------------------------------------------------- when I start my coaster service this is the output which is very strange: Start-coaster-service... Configuration: /home/iman/te/coaster-service.conf Service address: localhost Starting coaster-service Service port: 57175 Local port: 37589 Generating sites.xml *usage: ssh [-1246AaCfgKkMNnqsTtVvXxYy] [-b bind_address] [-c cipher_spec]* * [-D [bind_address:]port] [-e escape_char] [-F configfile]* * [-I pkcs11] [-i identity_file]* * [-L [bind_address:]port:host:hostport]* * [-l login_name] [-m mac_spec] [-O ctl_cmd] [-o option] [-p port] * * [-R [bind_address:]port:host:hostport] [-S ctl_path]* * [-W host:port] [-w local_tun[:remote_tun]]* * [user@]hostname [command]* *Starting worker on 216.47.142.127* *Generating config file* *iman at iman-ThinkPad-T420:~/te$ usage: ssh [-1246AaCfgKkMNnqsTtVvXxYy] [-b bind_address] [-c cipher_spec]* * [-D [bind_address:]port] [-e escape_char] [-F configfile]* * [-I pkcs11] [-i identity_file]* * [-L [bind_address:]port:host:hostport]* * [-l login_name] [-m mac_spec] [-O ctl_cmd] [-o option] [-p port] * * [-R [bind_address:]port:host:hostport] [-S ctl_path]* * [-W host:port] [-w local_tun[:remote_tun]]* * [user@]hostname [command]* and it never terminates until I type something on terminal. the output doesnt look right. after this step when I try running a swift simple swift code like this: iman at iman-ThinkPad-T420:~/te$ swift -sites.file sites.xml -tc.file tc catsn.swift Warning: Function toint is deprecated, at line 9 Swift trunk swift-r6010 cog-r3498 RunID: 20121108-2031-hhz64hqe Progress: time: Thu, 08 Nov 2012 20:31:23 -0600 Passive queue processor initialized. Callback URI is http://127.0.1.1:41681 Progress: time: Thu, 08 Nov 2012 20:31:53 -0600 Submitted:10 Progress: time: Thu, 08 Nov 2012 20:32:23 -0600 Submitted:10 and it never terminates, does anyone have an idea about it? I appreciate any comments or thoughts. Thanks Iman -- Iman Sadooghi Illinois Institute of Technology (IIT) Data-Intensive Distributed Systems Laboratory -------------- next part -------------- An HTML attachment was scrubbed... URL: From yanghao0614 at gmail.com Fri Nov 9 01:39:17 2012 From: yanghao0614 at gmail.com (Hao Yang) Date: Thu, 8 Nov 2012 23:39:17 -0800 Subject: [Swift-user] regarding Swift/Turbine compiler Message-ID: Hi, all: I am trying Swift/Turbine Compiler (STC). It seems that for foreach loops, the default mode is to run every iteration locally, if @async annotation is provided, then the iterations will be launched on other remote turbine engines, if there is app function inside a iteration, the iterations will be launched on workers. I am wondering what is the criteria for making these decisions (whether to run locally, on other engines or on workers) if the goal is to optimize the performance? If @unroll annotation is used to optimize performance, how can one choose a good unrolling factor? Thank you. -- Best Regards, Hao Yang Networked Systems Laboratory Electrical and Computer Engineering The University of British Columbia -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.g.armstrong at gmail.com Fri Nov 9 11:54:18 2012 From: tim.g.armstrong at gmail.com (Tim Armstrong) Date: Fri, 9 Nov 2012 11:54:18 -0600 Subject: [Swift-user] regarding Swift/Turbine compiler In-Reply-To: References: Message-ID: Hi Hao, Sorry for the confusion - which documentation are you looking at, out of curiosity? Basically, my advice would be to not worry about annotating the loops unless you are seeing performance problems. The default settings for foreach loops should give good performance in most use cases. By default they use a recursive loop splitting algorithm that is a good balance between parallelism and control-flow overhead. Here are a few rules of thumb about specific options: - The @unroll annotation is mainly useful if you have inter-iteration dependencies, as it can allow the compiler to optimize across loop iterations. E.g. in the example below it could save an array lookup by optimizing across iterations after unrolling: foreach x, i in A { B[i] = A[i] + A[i+1]; } - The @async annotation is rarely if ever useful: it tends to add additional overhead from more task spawns without meaningfully increasing parallelism, since any leaf functions within the loop body are spawned off as tasks regardless. - Tim On Fri, Nov 9, 2012 at 1:39 AM, Hao Yang wrote: > Hi, all: > > I am trying Swift/Turbine Compiler (STC). It seems that for foreach loops, > the default mode is to run every iteration locally, if @async annotation > is provided, then the iterations will be launched on other remote turbine > engines, if there is app function inside a iteration, the iterations will > be launched on workers. I am wondering what is the criteria for making > these decisions (whether to run locally, on other engines or on workers) if > the goal is to optimize the performance? If @unroll annotation is used to > optimize performance, how can one choose a good unrolling factor? > > Thank you. > > -- > Best Regards, > Hao Yang > Networked Systems Laboratory > Electrical and Computer Engineering > The University of British Columbia > > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yanghao0614 at gmail.com Fri Nov 9 16:58:00 2012 From: yanghao0614 at gmail.com (Hao Yang) Date: Fri, 9 Nov 2012 14:58:00 -0800 Subject: [Swift-user] regarding Swift/Turbine compiler In-Reply-To: References: Message-ID: Hi, Tim: I was reading Turbine paper, Swift/T guide website and some code. Thank you a lot for the clarification. In the example you gave, how to choose a good unrolling factor ? As greater unrolling factor can save more lookups but also generate greater code size and instructions. Another question is how to call external binaries inside app function? I didn't find information on the Swift/T guide website. Thank you. On Fri, Nov 9, 2012 at 9:54 AM, Tim Armstrong wrote: > Hi Hao, > Sorry for the confusion - which documentation are you looking at, out of > curiosity? > > Basically, my advice would be to not worry about annotating the loops > unless you are seeing performance problems. The default settings for > foreach loops should give good performance in most use cases. By default > they use a recursive loop splitting algorithm that is a good balance > between parallelism and control-flow overhead. > > Here are a few rules of thumb about specific options: > > - The @unroll annotation is mainly useful if you have inter-iteration > dependencies, as it can allow the compiler to optimize across loop > iterations. E.g. in the example below it could save an array lookup by > optimizing across iterations after unrolling: > > foreach x, i in A { > B[i] = A[i] + A[i+1]; > > } > > - The @async annotation is rarely if ever useful: it tends to add > additional overhead from more task spawns without meaningfully increasing > parallelism, since any leaf functions within the loop body are spawned off > as tasks regardless. > > - Tim > > On Fri, Nov 9, 2012 at 1:39 AM, Hao Yang wrote: > >> Hi, all: >> >> I am trying Swift/Turbine Compiler (STC). It seems that for foreach >> loops, the default mode is to run every iteration locally, if @async >> annotation is provided, then the iterations will be launched on other >> remote turbine engines, if there is app function inside a iteration, the >> iterations will be launched on workers. I am wondering what is the criteria >> for making these decisions (whether to run locally, on other engines or on >> workers) if the goal is to optimize the performance? If @unroll annotation >> is used to optimize performance, how can one choose a good unrolling >> factor? >> >> Thank you. >> >> -- >> Best Regards, >> Hao Yang >> Networked Systems Laboratory >> Electrical and Computer Engineering >> The University of British Columbia >> >> >> >> _______________________________________________ >> Swift-user mailing list >> Swift-user at ci.uchicago.edu >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user >> > > -- Best Regards, Hao Yang Networked Systems Laboratory Electrical and Computer Engineering The University of British Columbia -------------- next part -------------- An HTML attachment was scrubbed... URL: From iraicu at cs.iit.edu Sat Nov 10 09:08:46 2012 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Sat, 10 Nov 2012 09:08:46 -0600 Subject: [Swift-user] CFP: IEEE/ACM CCGrid 2013 -- deadline extension to November 22 Message-ID: <509E6DFE.6040705@cs.iit.edu> **** CALL FOR PAPERS **** The 13th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid 2013) Delft University of Technology, Delft, the Netherlands May 13-16, 2013 http://www.pds.ewi.tudelft.nl/ccgrid2013 ********************************************************************** ******* Extension of paper deadline until November 22, 2012 ****** ********************************************************************** ***** The CCGrid 2013 Workshops have been posted on the website ***** Rapid advances in architectures, networks, and systems and middleware technologies are leading to new concepts in and platforms for computing, ranging from Clusters and Grids to Clouds and Datacenters. CCGrid is a series of very successful conferences, sponsored by the IEEE Computer Society Technical Committee on Scalable Computing (TCSC) and the ACM, with the overarching goal of bringing together international researchers, developers, and users to provide an international forum to present leading research activities and results on a broad range of topics related to these concepts and platforms, and their applications. The conference features keynotes, technical presentations, workshops, tutorials, and posters, as well as the SCALE challenge featuring live demonstrations. In 2013, CCGrid will come to the Netherlands for the first time, and will be held in Delft, a historical, picturesque city that is less than one hour away from Amsterdam-Schiphol airport. The main conference will be held on May 14-16 (Tuesday to Thursday), with tutorials and affiliated workshops taking place on May 13 (Monday). **** IMPORTANT DATES **** Papers Due: 22 November 2012 Anywhere on Earth (extended) Author Notifications: 24 January 2013 Final Papers Due: 22 February 2013 **** TOPICS OF INTEREST **** CCGrid 2013 will have a focus on important and immediate issues that are significantly influencing all aspects of cluster, cloud and grid computing. Topics of interest include, but are not limited to: * Applications and Experiences: Applications to real and complex problems in science, engineering, business, and society; User studies; Experiences with large-scale deployments, systems, or applications * Architecture: System architectures, design and deployment; Power and cooling; Security and reliability; High availability solutions * Autonomic Computing and Cyberinfrastructure: Self-managed behavior, models and technologies; Autonomic paradigms and systems (control-based, bio-inspired, emergent, etc.); Bio-inspired optimizations and computing * Cloud Computing: Cloud architectures; Software tools and techniques for clouds * Multicore and Accelerator-based Computing: Software and application techniques to utilize multicore architectures and accelerators in clusters, grids, and clouds * Performance Modeling and Evaluation: Performance prediction and modeling; Monitoring and evaluation tools; Analysis of system and application performance; Benchmarks and testbeds * Programming Models, Systems, and Fault-Tolerant Computing: Programming models and environments for cluster, cloud, and grid computing; Fault-tolerant systems, programs and algorithms; Systems software to support efficient computing * Scheduling and Resource Management: Techniques to schedule jobs and resources on cluster, cloud, and grid computing platforms; SLA definition and enforcement **** PAPER SUBMISSION GUIDELINES **** Authors are invited to submit papers electronically in PDF format. Submitted manuscripts should be structured as technical papers and may not exceed 8 letter-size (8.5 x 11) pages including figures, tables and references using the IEEE format for conference proceedings. Submissions not conforming to these guidelines may be returned without review. Authors should make sure that their file will print on a printer that uses letter-size (8.5 x 11) paper. The official language of the conference is English. All manuscripts will be reviewed and will be judged on correctness, originality, technical strength, significance, quality of presentation, and interest and relevance to the conference attendees. Submitted papers must represent original unpublished research that is not currently under review for any other conference or journal. Papers not following these guidelines will be rejected without review and further action may be taken, including (but not limited to) notifications sent to the heads of the institutions of the authors and sponsors of the conference. Submissions received after the due date, exceeding the page limit, or not appropriately structured may not be considered. Authors may contact the conference chairs for more information. The proceedings will be published through the IEEE Computer Society Press, USA, and will be made available online through the IEEE Digital Library. **** CALL FOR TUTORIAL AND WORKSHOP PROPOSALS **** Tutorials and workshops affiliated with CCGrid 2013 will be held on May 13 (Monday). For more information on the tutorials and workshops and for the complete Call for Tutorial and Workshop Proposals, please see the conference website. **** GENERAL CHAIR **** Dick Epema, Delft University of Technology, the Netherlands **** PROGRAM CHAIR **** Thomas Fahringer, University of Innsbruck, Austria **** PROGRAM VICE-CHAIRS **** Rosa Badia, Barcelona Supercomputing Center, Spain Henri Bal, Vrije Universiteit, the Netherlands Marios Dikaiakos, University of Cyprus, Cyprus Kirk Cameron, VirginiaTech, USA Daniel Katz, University of Chicago & Argonne Nat Lab, USA Kate Keahey, Argonne National Laboratory, USA Martin Schulz, Lawrence Livermore National Laboratory, USA Douglas Thain, University of Notre Dame, USA Cheng-Zhong Xu, Shenzhen Inst. of Advanced Techn, China **** WORKSHOPS CO-CHAIRS **** Shantenu Jha, Rutgers and Louisana State University, USA Ioan Raicu, Illinois Institute of Technology, USA **** TOTORIALS CHAIR **** Radu Prodan, University of Innsbruck, Austria **** DOCTORAL SYMPOSIUM CO-CHAIRS **** Yogesh Simmhan, University of Southern California, USA Ana Varbanescu, Delft Univ of Technology, the Netherlands **** SUBMISSIONS AND PROCEEDINGS CHAIR **** Pavan Balaji, Argonne National Laboratory, USA **** FINANCE AND REGISTRATION CHAIR **** Alexandru Iosup, Delft Univ of Technology, the Netherlands **** PUBLICITY CHAIRS **** Nazareno Andrade, University Federal de Campina Grance, Brazil Gabriel Antoniu, INRIA, France Bahman Javadi, University of Western Sysney, Australia Ioan Raicu, Illinois Institute of Technology and Argonne National Laboratory, USA Kin Choong Yow, Shenzhen Inst. of Advanced Technology, China **** CYBER CHAIR **** Stephen van der Laan, Delft University of Technology, the Netherlands -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ ================================================================= ================================================================= From marialemos72 at gmail.com Fri Nov 16 11:33:37 2012 From: marialemos72 at gmail.com (WorldCIST) Date: Fri, 16 Nov 2012 17:33:37 +0000 Subject: [Swift-user] Best papers published in JCR/ISI JOURNALS - Deadline: November 25 Message-ID: <20121116173257.0BA497CC0E9@mailrelay.anl.gov> Apologies if you are receiving this mail more than once... Please disseminate by colleagues, researchers, students, etc. Thanks a lot! ********************************************************************************** WorldCIST'13 The 2013 World Conference on Information Systems and Technologies March 27 - 30, Algarve, Portugal http://www.aisti.eu/worldcist13/ ********************************************************************************** SCOPE The 2013 World Conference on Information Systems and Technologies (WorldCIST'13: http://www.aisti.eu/worldcist13/) is a global forum for researchers and practitioners to present and discuss the most recent innovations, trends, results, experiences and concerns in the several perspectives of Information Systems and Technologies. We are pleased to invite you to submit your papers to WorldCISTI'13. All submissions will be reviewed on the basis of relevance, originality, importance and clarity. THEMES Submitted papers should be related with one or more of the main themes proposed for the Conference: A) Information and Knowledge Management (IKM); B) Organizational Models and Information Systems (OMIS); C) Intelligent and Decision Support Systems (IDSS); D) Software Systems, Architectures, Applications and Tools (SSAAT); E) Computer Networks, Mobility and Pervasive Systems (CNMPS); F) Human-Computer Interaction (HCI). TYPES OF SUBMISSIONS AND DECISIONS Four types of papers can be submitted: Full paper: Finished or consolidated R&D works, to be included in one of the Conference themes. These papers are assigned a 10-page limit. Short paper: Ongoing works with relevant preliminary results, open to discussion. These papers are assigned a 7-page limit. Poster paper: Initial work with relevant ideas, open to discussion. These papers are assigned to a 4-page limit. Company paper: Companies' papers that show practical experience, R & D, tools, etc., focused on some topics of the conference. These papers are assigned to a 4-page limit. Submitted papers must comply with the format of Advances in Intelligent Systems and Computing Series (see http://www.aisti.eu/worldcist13/springerformat.doc) be written in English, must not have been published before, not be under review for any other conference or publication and not include any information leading to the authors? identification. Therefore, the authors? names, affiliations and bibliographic references should not be included in the version for evaluation by the Program Committee. This information should only be included in the camera-ready version. All papers will be subjected to a ?double-blind review? by at least two members of the Program Committee. Based on Program Committee evaluation, a paper can be rejected or accepted by the Conference Chairs. In the later case, it can be accepted as the type originally submitted or as another type. Thus, full papers can be accepted as short papers or poster papers only. Similarly, short papers can be accepted as poster papers only. In these cases, the authors will be allowed to maintain the original number of pages in the camera-ready version. The authors of accepted poster papers must also build and print a poster to be exhibited during the Conference. This poster must follow an A1 or A2 vertical format. The Conference includes Work Sessions where these posters are presented and orally discussed, with a 5 minute limit per poster. The authors of accepted full papers will have 15 minutes to present their work in a Conference Work Session; approximately 5 minutes of discussion will follow each presentation. The authors of accepted short papers and company papers will have 11 minutes to present their work in a Conference Work Session; approximately 4 minutes of discussion will follow each presentation. PUBLICATION AND INDEXING To ensure that a full paper, short paper, poster paper or company paper is published in the Proceedings, at least one of the authors must be fully registered by the 11th of January 2013, and the paper must comply with the suggested layout and page-limit. Additionally, all recommended changes must be addressed by the authors before they submit the camera-ready version. No more than one paper per registration will be published in the Conference Proceedings. An extra fee must be paid for publication of additional papers, with a maximum of two additional papers per registration. Full and short papers will be published in Proceedings by Springer, in Advances in Intelligent Systems and Computing Series. Poster and company papers will be published in Proceedings by AISTI. Published full and short papers will be indexed by ISI, EI-Compendex, SCOPUS, DBLP and EBSCO, among others, and will be available in the SpringerLink Digital Library. Published poster and company papers will be indexed in EI-Compendex and EBSCO. The authors of the best selected papers will be invited to extend them for publication in edited books and in international journals indexed by ISI/JCR, SCOPUS and/or DBLP, among others, such as: - ACM Transactions on Modeling and Computer Simulation (TOMACS) - Online Information Review (OIR) - Informatics for Health and Social Care (IHSC) - Computer Science and Information Systems (ComSIS) - Telecommunication Systems Journal (TSJ) - INFORMATION - An International Interdisciplinary Journal - Journal of Organizational and End User Computing (JOEUC) - Information Researh (IR) - International Journal of Internet Protocol Technology (IJIPT) - Studies in Computational Intelligence (SCI) - Journal of Advanced Computational Intelligence and Intelligent Informatics (JACIII) - Journal of Electrical and Computer Engineering (JECE): Special Issue in Advances in Radar Technology - WSEAS Transactions on Systems (TS) - Library Review (LR) - Education for Information (EI) - International Journal of IT/Business Alignment and Governance (IJITBAG) - International Journal of Systems and Service-Oriented Engineering (IJSSOE) - International Journal of Interactive Multimedia and Artificial Intelligence (IJIMAI) IMPORTANT DATES Paper Submission: November 25, 2012 Notification of Acceptance: December 30, 2012 Camera-ready Submission: January 9, 2013 Payment of Registration, to ensure the inclusion of an accepted paper in the conference proceedings: January 11, 2013. - Kind regards, Maria Lemos WorldCIST'13 http://www.aisti.eu/worldcist13/ From iraicu at cs.iit.edu Sun Nov 18 19:27:18 2012 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Sun, 18 Nov 2012 19:27:18 -0600 Subject: [Swift-user] Fwd: [DistComp] CFP: 2nd IEEE International Workshop on Workflow Models, Systems, Services and Applications in the Cloud (CloudFlow) 2013 In-Reply-To: References: Message-ID: <50A98AF6.2090602@cs.iit.edu> This seems relevant to the Swift community! Ioan -------- Original Message -------- Subject: [DistComp] CFP: 2nd IEEE International Workshop on Workflow Models, Systems, Services and Applications in the Cloud (CloudFlow) 2013 Date: Mon, 19 Nov 2012 09:22:18 +0800 From: yong zhao To: distributed-computing-announce at datasys.cs.iit.edu *Second IEEE International Workshop on Workflow Models, Systems, Services and Applications in the Cloud (CloudFlow) 2013* /To be held in conjunction with the 27th IEEE International Parallel & Distributed Processing Symposium (IPDPS) 2013, Cambridge, Boston, Massachusetts, USA, May 20-24, 2013./ http://www.cloud-uestc.cn/cloudflow/home.html *Overview* Cloud computing is gaining tremendous momentum in both academia and industry, more and more people are migrating their data and applications into the Cloud. We have observed wide adoption of the MapReduce computing model and the open source Hadoop system for large scale distributed data processing, and a variety of ad hoc mashup techniques that weave together Web applications. However, these are just first steps towards managing complex task and data dependencies in the Cloud, as there are more challenging issues such as large parameter space exploration, data partitioning and distribution, scheduling and optimization, smart reruns, and provenance tracking associated with workflow execution. Cloud needs structured and mature workflow technologies to handle such issues, and vice versa, as Cloud offers unprecedented scalability to workflow systems, and could potentially change the way we perceive and conduct research and experiments. The scale and complexity of the science and data analytics problems that can be handled can be greatly increased on the Cloud, and the on-demand nature of resource allocation on the Cloud will also help improve resource utilization and user experience. As Cloud computing provides a paradigm-shifting utility-oriented computing model in terms of the unprecedented size of datacenter-level resource pool and the on-demand resource provisioning mechanism, there are lots of challenges in bringing Cloud and workflows together. We need high level languages and computing models for large scale workflow specification; we need to adapt existing workflow architectures into the Cloud, and integrate workflow systems with Cloud infrastructure and resources; we also need to leverage Cloud data storage technologies to efficiently distribute data over a large number of nodes and explore data locality during computation etc. We organize the CloudFlow workshop as a venue for the workflow and Cloud communities to define models and paradigms, present their state-of-the-art work, share their thoughts and experiences, and explore new directions in realizing workflows in the Cloud. *Topics:* We welcome the submission of original work related to the topics listed below, which include (in the context of Cloud): ? Models and Languages for Large Scale Workflow Specification ? Workflow Architecture and Framework ? Large Scale Workflow Systems ? Service Workflow ? Workflow Composition and Orchestration ? Workflow Migration into the Cloud ? Workflow Scheduling and Optimization ? Cloud Middleware in Support of Workflow ? Virtualized Environment ? Workflow Applications and Case Studies ? Performance and Scalability Analysis ? Peta-Scale Data Processing ? Event Processing and Messaging ? Real-Time Analytics ? Provenance *Paper Submission* Authors are invited to submit papers with unpublished, original work. The papers should not exceed 10 single-spaced double-column pages using 10-point size font on 8.5x11 inch pages (IEEE conference style), including figures, tables, and references. Paper submission should be done via the online CMT system, Microsoft?s Academic Conference Management Service (*https://cmt.research.microsoft.com/CF2013*) by midnight January 9th, 2013 Pacific Time. The final format should be in PDF. Proceedings of the workshop will be published by the IEEE Digital Library (indexed by EI) and distributed at the conference. Selected excellent work may be eligible for additional post-conference publication as journal articles or book chapters. Submission implies the willingness of at least one of the authors to register and present the paper. *Important Dates* ** Paper submission: January 9th, 2013 Acceptance notification: February 8th, 2013 Final paper due: Feb 19th, 2013 *Organization* Workshop Chairs: Dr. Yong Zhao University of Electronic Science and Technology of China, China yongzh04 at gmail.com Dr. Cui Lin California State University, Fresno, USA clin at csufresno.edu Dr. Shiyong Lu Wayne State University, USA shiyong at wayne.edu Program Chair: Dr. Wenhong Tian University of Electronic Science and Technology of China, China Publicity Chair: Dr. Ruini Xue University of Electronic Science and Technology of China, China *Steering Committee * ? Daniel S. Katz, University of Chicago, U.S.A. ? Mike Wilde, University of Chicago, U.S.A. ? Ewa Deelman, University of South California, U.S.A. ? Tevfik Kosar, University at Buffalo, U.S.A. ? Ilkay Altintas, San Diego Supercomputer Center, U.S.A. ? Ioan Raicu, Illinois Institute of Technology, U.S.A. ? Yogesh Simmhan, University of Southern California, U.S.A. ? Ian Taylor, Cardiff University, U.K. ? Weimin Zheng, Tsinghua University, China ? Hai Jin, Huazhong University of Science and Engineering, China ? Wanchun Dou, Nanjing University, China ? Hui Zhang, National Science and Technology Infrastructure, China *Program Committee * ? Shawn Bowers, Gonzaga University, U.S.A. ? Douglas Thain, University of Notre Dame, U.S.A. ? Ian Gorton, Pacific Northwest National Laboratory, U.S.A. ? Artem Chebotko, University of Texas at Pan American, U.S.A. ? Weisong Shi, Wayne State University, U.S.A. ? Paolo Missier, Newcastle University, U.K. ? Wei Tan, IBM T. J. Watson Research Center, U.S.A. ? Jianwu Wang, San Diego Super Computer Center, U.S.A. ? Ping Yang, Binghamton University, U.S.A. ? Jian Guo, Harvard University, U.S.A. ? Liqiang Wang, University of Wyoming, U.S.A. ? Paul Groth, VU University Amsterdam, the Netherlands ? Zhiming Zhao, University of Amsterdam, the Netherlands ? Marta Mattoso, Federal University of Rio de Janeiro, Brazil ? Wenhong Tian, University of Electronic Science and Technology of China, China ? Ruini Xue, Tsinghua University, China ? Jian Cao, Shanghai Jiaotong University, China ? Jianxun Liu, Hunan University of Science and Technology, China ? Song Zhang, Chinese Academy of Sciences, China ? Hua Hu, Hangzhou Dianzi University, China ========================================================== Yong Zhao yongzh04 at gmail.com Director, Extreme Scale Network Computing and Service Laboratory Professor, School of Computer Science and Engineering University of Electronic Science and Technology of China http://cloud-uestc.cn ========================================================== -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- _______________________________________________ distributed-computing-announce mailing list distributed-computing-announce at datasys.cs.iit.edu http://datasys.cs.iit.edu/mailman/listinfo/distributed-computing-announce From yanghao0614 at gmail.com Tue Nov 20 19:22:45 2012 From: yanghao0614 at gmail.com (Hao Yang) Date: Tue, 20 Nov 2012 17:22:45 -0800 Subject: [Swift-user] assign temporary file to mapper Message-ID: Hi, all: I have some problem assigning temporary file to some mapper variable (explained in the code below). type RSFile; string output_name; output_name = @arg("out","out.rsf"); RSFile output_data ; RSFile output_chunks[]; // some calculation to compute output_chunks[0], output_chunks[1], output_chunks[2] // at the end output_data = output_chunks[2]; When I try this script, swift reports "File output_chunks[2] not found " error and failed. But the output_chunks[2] is still available in the directory generated by swift (_concurrent/output_chunks- ae9bd0b3-ad88-45b0-95f4-f036d2d70e58--array). Should this error message be expected? Thank you. -- Best Regards, Hao Yang The University of British Columbia -------------- next part -------------- An HTML attachment was scrubbed... URL: From wilde at mcs.anl.gov Tue Nov 20 20:29:27 2012 From: wilde at mcs.anl.gov (Michael Wilde) Date: Tue, 20 Nov 2012 20:29:27 -0600 (CST) Subject: [Swift-user] assign temporary file to mapper In-Reply-To: Message-ID: <1382635511.48227.1353464967314.JavaMail.root@zimbra.anl.gov> Hao, Are you sure you assigned a value to output_chunks[2]? Can you post the entire program, and the entire error message? Thanks, - Mike ----- Original Message ----- > From: "Hao Yang" > To: swift-user at ci.uchicago.edu > Cc: mosastore at googlegroups.com > Sent: Tuesday, November 20, 2012 7:22:45 PM > Subject: [Swift-user] assign temporary file to mapper > Hi, all: > > > I have some problem assigning temporary file to some mapper variable > (explained in the code below). > > > type RSFile; > string output_name; > output_name = @arg("out","out.rsf"); > RSFile output_data ; > RSFile output_chunks[]; > // some calculation to compute output_chunks[0], output_chunks[1], > output_chunks[2] > // at the end > output_data = output_chunks[2]; > > > > When I try this script, swift reports " File output_chunks[2] not > found " error and failed. But the output_chunks[2] is still available > in the directory generated by swift ( _ concurrent/output_chunks- > ae9bd0b3-ad88-45b0-95f4- f036d2d70e58--array). Should this error > message be expected? > > > Thank you. > > -- > Best Regards, > Hao Yang > > The University of British Columbia > > > > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory From yanghao0614 at gmail.com Tue Nov 20 22:33:15 2012 From: yanghao0614 at gmail.com (Hao Yang) Date: Tue, 20 Nov 2012 20:33:15 -0800 Subject: [Swift-user] assign temporary file to mapper In-Reply-To: <1382635511.48227.1353464967314.JavaMail.root@zimbra.anl.gov> References: <1382635511.48227.1353464967314.JavaMail.root@zimbra.anl.gov> Message-ID: Hi, Michael: The output_chunks[2] is assigned in the iterate via output_chunks[i] = appAddFiles(processed_chunks[i], output_chunks[i-1]). (I put the script code and the error message below). *// swift code * type RSFile; // rsf file type StringFile; // File that stores strings // APPs // getNDims: Extract last dimension from rsf file app (StringFile ndims) appGetNDims(RSFile input) { sffiledims_script @filename(input) "1" stdout=@filename(ndims); } // getDims: Extract dimension list from rsf file app (StringFile dims) appGetDims(RSFile input) { sffiledims_script @filename(input) "2" stdout=@filename(dims); } // getWindow: Window a slice from rsf file app (RSFile out) appSplitFile(RSFile inp, string dim, int slice) { sfwindow_script @filename(inp) @filename(out) dim slice; } // addFiles: adds inp1 and inp2 and returns out app (RSFile out) appAddFiles(RSFile inp1, RSFile inp2) { sfadd_script @filename(out) @filename(inp1) @filename(inp2); } // In-core variables string input_name; // input name of the file string output_name; // output name of the file string last_dim; // Last dimension of file (default splitting dimension) string split_dim; // Splitting dimension string dims_list; // string of dimensions (unsplitted) string dims_array[]; // Array of string of dimensions (splitted) int dim_length; // length of splitted dimension // File variables // Setup input and output files RSFile input_data ; RSFile output_data ; StringFile ndims_file; // number of dimensions of the input file StringFile dims_file; // dimension array of the input file RSFile input_chunks[]; // Splitted input RSFile processed_chunks[]; // data chunks after parallel processing RSFile output_chunks[]; // Temporary output array for reduction // main part // Define and parse input arguments input_name = @arg("in","in.rsf"); output_name = @arg("out","out.rsf"); // extract dims and setup variables ndims_file = appGetNDims(input_data); dims_file = appGetDims(input_data); // Define and parse split dimension and dimensions array last_dim = readData(ndims_file); split_dim = @arg("dim", last_dim); dims_list = readData(dims_file); dims_array = @strsplit(dims_list,","); dim_length = @toint(dims_array[@toint(split_dim)-1]); trace("Dim length = ", dim_length); // Split the files parallelly foreach i in [0:dim_length-1] { trace("Splitting ", i); input_chunks[i] = appSplitFile(input_data, split_dim, i+1); trace("Working furiously on ", @filename(input_chunks[i])); trace("Chunk number: ", i); processed_chunks[i] = input_chunks[i]; } // Add all the chunks together iterate i { if( i == 0 ) { trace("Setting chunk 0 to ", @filename(output_chunks[0])); output_chunks[0] = processed_chunks[0]; } else { trace("Adding ", i); output_chunks[i] = appAddFiles(processed_chunks[i], output_chunks[i-1]); } } until (i == dim_length-1); output_data = output_chunks[dim_length-1]; *// swift message* RunID: 20121120-1538-mt4hjjd6 Progress: time: Tue, 20 Nov 2012 15:38:03 -0800 SwiftScript trace: Setting chunk 0 to , _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-0 SwiftScript trace: filename of output_chunks[0] is, _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-0 SwiftScript trace: filename of processed_chunks[0] is, _concurrent/processed_chunks-824d746f-2a4d-48b6-8b51-e2534f1f86ca--array//elt-0 Progress: time: Tue, 20 Nov 2012 15:38:04 -0800 Submitting:1 Finished successfully:1 SwiftScript trace: Dim length = , 3.0 SwiftScript trace: Splitting , 1.0 SwiftScript trace: Chunk number: , 1.0 SwiftScript trace: Chunk number: , 2.0 SwiftScript trace: Working furiously on , _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-1 SwiftScript trace: Splitting , 0.0 SwiftScript trace: Chunk number: , 0.0 SwiftScript trace: Splitting , 2.0 SwiftScript trace: Working furiously on , _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-2 SwiftScript trace: Working furiously on , _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-0 Progress: time: Tue, 20 Nov 2012 15:38:05 -0800 Selecting site:1 Stage in:1 Finished successfully:3 Progress: time: Tue, 20 Nov 2012 15:38:06 -0800 Checking status:1 Finished successfully:4 SwiftScript trace: Adding , 1.0 SwiftScript trace: filename of output_chunks[i] is, _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-1 SwiftScript trace: filename of output_chunks[i-1] is, _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-0 SwiftScript trace: filename of processed_chunks[i] is, _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-1 SwiftScript trace: Adding , 2.0 SwiftScript trace: filename of processed_chunks[i] is, _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-2 SwiftScript trace: filename of output_chunks[i] is, _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-2 SwiftScript trace: filename of output_chunks[i-1] is, _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-1 Execution failed: File not found: /users/slic/tlai/work/mapReduceScripts/./_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array/_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array/elt-2 >From the trace we can see filename of output_chunks[2] is _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-2 (we also verified the file is there after the execution failure). Thank you. On Tue, Nov 20, 2012 at 6:29 PM, Michael Wilde wrote: > Hao, > > Are you sure you assigned a value to output_chunks[2]? > > Can you post the entire program, and the entire error message? > > Thanks, > > - Mike > > > ----- Original Message ----- > > From: "Hao Yang" > > To: swift-user at ci.uchicago.edu > > Cc: mosastore at googlegroups.com > > Sent: Tuesday, November 20, 2012 7:22:45 PM > > Subject: [Swift-user] assign temporary file to mapper > > Hi, all: > > > > > > I have some problem assigning temporary file to some mapper variable > > (explained in the code below). > > > > > > type RSFile; > > string output_name; > > output_name = @arg("out","out.rsf"); > > RSFile output_data ; > > RSFile output_chunks[]; > > // some calculation to compute output_chunks[0], output_chunks[1], > > output_chunks[2] > > // at the end > > output_data = output_chunks[2]; > > > > > > > > When I try this script, swift reports " File output_chunks[2] not > > found " error and failed. But the output_chunks[2] is still available > > in the directory generated by swift ( _ concurrent/output_chunks- > > ae9bd0b3-ad88-45b0-95f4- f036d2d70e58--array). Should this error > > message be expected? > > > > > > Thank you. > > > > -- > > Best Regards, > > Hao Yang > > > > The University of British Columbia > > > > > > > > > > > > _______________________________________________ > > Swift-user mailing list > > Swift-user at ci.uchicago.edu > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > -- > Michael Wilde > Computation Institute, University of Chicago > Mathematics and Computer Science Division > Argonne National Laboratory > > -- Best Regards, Hao Yang Networked Systems Laboratory Electrical and Computer Engineering The University of British Columbia -------------- next part -------------- An HTML attachment was scrubbed... URL: From marialemos72 at gmail.com Wed Nov 21 11:44:29 2012 From: marialemos72 at gmail.com (WorldCIST) Date: Wed, 21 Nov 2012 17:44:29 +0000 Subject: [Swift-user] WorldCIST'13: Indexed by ISI, SCOPUS, DBLP and EI-Compendex - Deadline: November 28 Message-ID: <20121121174345.D8D6D7CC0D6@mailrelay.anl.gov> Apologies if you are receiving this mail more than once... Please disseminate by colleagues, researchers, students, etc. Thanks a lot! ********************************************************************************** WorldCIST'13 The 2013 World Conference on Information Systems and Technologies March 27 - 30, Algarve, Portugal http://www.aisti.eu/worldcist13/ ********************************************************************************** SCOPE The 2013 World Conference on Information Systems and Technologies (WorldCIST'13: http://www.aisti.eu/worldcist13/) is a global forum for researchers and practitioners to present and discuss the most recent innovations, trends, results, experiences and concerns in the several perspectives of Information Systems and Technologies. We are pleased to invite you to submit your papers to WorldCISTI'13. All submissions will be reviewed on the basis of relevance, originality, importance and clarity. THEMES Submitted papers should be related with one or more of the main themes proposed for the Conference: A) Information and Knowledge Management (IKM); B) Organizational Models and Information Systems (OMIS); C) Intelligent and Decision Support Systems (IDSS); D) Software Systems, Architectures, Applications and Tools (SSAAT); E) Computer Networks, Mobility and Pervasive Systems (CNMPS); F) Human-Computer Interaction (HCI). TYPES OF SUBMISSIONS AND DECISIONS Four types of papers can be submitted: Full paper: Finished or consolidated R&D works, to be included in one of the Conference themes. These papers are assigned a 10-page limit. Short paper: Ongoing works with relevant preliminary results, open to discussion. These papers are assigned a 7-page limit. Poster paper: Initial work with relevant ideas, open to discussion. These papers are assigned to a 4-page limit. Company paper: Companies' papers that show practical experience, R & D, tools, etc., focused on some topics of the conference. These papers are assigned to a 4-page limit. Submitted papers must comply with the format of Advances in Intelligent Systems and Computing Series (see http://www.aisti.eu/worldcist13/springerformat.doc) be written in English, must not have been published before, not be under review for any other conference or publication and not include any information leading to the authors? identification. Therefore, the authors? names, affiliations and bibliographic references should not be included in the version for evaluation by the Program Committee. This information should only be included in the camera-ready version. All papers will be subjected to a ?double-blind review? by at least two members of the Program Committee. Based on Program Committee evaluation, a paper can be rejected or accepted by the Conference Chairs. In the later case, it can be accepted as the type originally submitted or as another type. Thus, full papers can be accepted as short papers or poster papers only. Similarly, short papers can be accepted as poster papers only. In these cases, the authors will be allowed to maintain the original number of pages in the camera-ready version. The authors of accepted poster papers must also build and print a poster to be exhibited during the Conference. This poster must follow an A1 or A2 vertical format. The Conference includes Work Sessions where these posters are presented and orally discussed, with a 5 minute limit per poster. The authors of accepted full papers will have 15 minutes to present their work in a Conference Work Session; approximately 5 minutes of discussion will follow each presentation. The authors of accepted short papers and company papers will have 11 minutes to present their work in a Conference Work Session; approximately 4 minutes of discussion will follow each presentation. PUBLICATION AND INDEXING To ensure that a full paper, short paper, poster paper or company paper is published in the Proceedings, at least one of the authors must be fully registered by the 11th of January 2013, and the paper must comply with the suggested layout and page-limit. Additionally, all recommended changes must be addressed by the authors before they submit the camera-ready version. No more than one paper per registration will be published in the Conference Proceedings. An extra fee must be paid for publication of additional papers, with a maximum of two additional papers per registration. Full and short papers will be published in Proceedings by Springer, in Advances in Intelligent Systems and Computing Series. Poster and company papers will be published in Proceedings by AISTI. Published full and short papers will be indexed by ISI, EI-Compendex, SCOPUS, DBLP and EBSCO, among others, and will be available in the SpringerLink Digital Library. Published poster and company papers will be indexed in EI-Compendex and EBSCO. The authors of the best selected papers will be invited to extend them for publication in edited books and in international journals indexed by ISI/JCR, SCOPUS and/or DBLP, among others, such as: - ACM Transactions on Modeling and Computer Simulation (TOMACS) - Online Information Review (OIR) - Informatics for Health and Social Care (IHSC) - Computer Science and Information Systems (ComSIS) - Telecommunication Systems Journal (TSJ) - INFORMATION - An International Interdisciplinary Journal - Journal of Organizational and End User Computing (JOEUC) - Information Researh (IR) - International Journal of Internet Protocol Technology (IJIPT) - Studies in Computational Intelligence (SCI) - Journal of Advanced Computational Intelligence and Intelligent Informatics (JACIII) - Journal of Electrical and Computer Engineering (JECE): Special Issue in Advances in Radar Technology - WSEAS Transactions on Systems (TS) - Library Review (LR) - Education for Information (EI) - International Journal of IT/Business Alignment and Governance (IJITBAG) - International Journal of Systems and Service-Oriented Engineering (IJSSOE) - International Journal of Interactive Multimedia and Artificial Intelligence (IJIMAI) IMPORTANT DATES Paper Submission: November 25, 2012 Notification of Acceptance: December 30, 2012 Camera-ready Submission: January 9, 2013 Payment of Registration, to ensure the inclusion of an accepted paper in the conference proceedings: January 11, 2013. - Kind regards, Maria Lemos WorldCIST'13 http://www.aisti.eu/worldcist13/ From marialemos72 at gmail.com Sun Nov 25 18:19:52 2012 From: marialemos72 at gmail.com (WorldCIST) Date: Mon, 26 Nov 2012 00:19:52 +0000 Subject: [Swift-user] WorldCIST'13: Indexed by ISI, SCOPUS, DBLP, EI, etc. - Extended deadline: December 7 Message-ID: <20121126001908.867887CC094@mailrelay.anl.gov> Apologies if you are receiving this mail more than once... Please disseminate by colleagues, researchers, students, etc. Thanks a lot! ********************************************************************************** WorldCIST'13 The 2013 World Conference on Information Systems and Technologies March 27 - 30, Algarve, Portugal http://www.aisti.eu/worldcist13/ ********************************************************************************** SCOPE The 2013 World Conference on Information Systems and Technologies (WorldCIST'13: http://www.aisti.eu/worldcist13/) is a global forum for researchers and practitioners to present and discuss the most recent innovations, trends, results, experiences and concerns in the several perspectives of Information Systems and Technologies. We are pleased to invite you to submit your papers to WorldCISTI'13. All submissions will be reviewed on the basis of relevance, originality, importance and clarity. THEMES Submitted papers should be related with one or more of the main themes proposed for the Conference: A) Information and Knowledge Management (IKM); B) Organizational Models and Information Systems (OMIS); C) Intelligent and Decision Support Systems (IDSS); D) Software Systems, Architectures, Applications and Tools (SSAAT); E) Computer Networks, Mobility and Pervasive Systems (CNMPS); F) Radar Technologies (RAT); G) Human-Computer Interaction (HCI). TYPES OF SUBMISSIONS AND DECISIONS Four types of papers can be submitted: Full paper: Finished or consolidated R&D works, to be included in one of the Conference themes. These papers are assigned a 10-page limit. Short paper: Ongoing works with relevant preliminary results, open to discussion. These papers are assigned a 7-page limit. Poster paper: Initial work with relevant ideas, open to discussion. These papers are assigned to a 4-page limit. Company paper: Companies' papers that show practical experience, R & D, tools, etc., focused on some topics of the conference. These papers are assigned to a 4-page limit. Submitted papers must comply with the format of Advances in Intelligent Systems and Computing Series (see http://www.aisti.eu/worldcist13/springerformat.doc) be written in English, must not have been published before, not be under review for any other conference or publication and not include any information leading to the authors? identification. Therefore, the authors? names, affiliations and bibliographic references should not be included in the version for evaluation by the Program Committee. This information should only be included in the camera-ready version. All papers will be subjected to a ?double-blind review? by at least two members of the Program Committee. Based on Program Committee evaluation, a paper can be rejected or accepted by the Conference Chairs. In the later case, it can be accepted as the type originally submitted or as another type. Thus, full papers can be accepted as short papers or poster papers only. Similarly, short papers can be accepted as poster papers only. In these cases, the authors will be allowed to maintain the original number of pages in the camera-ready version. The authors of accepted poster papers must also build and print a poster to be exhibited during the Conference. This poster must follow an A1 or A2 vertical format. The Conference includes Work Sessions where these posters are presented and orally discussed, with a 5 minute limit per poster. The authors of accepted full papers will have 15 minutes to present their work in a Conference Work Session; approximately 5 minutes of discussion will follow each presentation. The authors of accepted short papers and company papers will have 11 minutes to present their work in a Conference Work Session; approximately 4 minutes of discussion will follow each presentation. PUBLICATION AND INDEXING To ensure that a full paper, short paper, poster paper or company paper is published in the Proceedings, at least one of the authors must be fully registered by the 11th of January 2013, and the paper must comply with the suggested layout and page-limit. Additionally, all recommended changes must be addressed by the authors before they submit the camera-ready version. No more than one paper per registration will be published in the Conference Proceedings. An extra fee must be paid for publication of additional papers, with a maximum of two additional papers per registration. Full and short papers will be published in Proceedings by Springer, in Advances in Intelligent Systems and Computing Series. Poster and company papers will be published in Proceedings by AISTI. Published full and short papers will be indexed by ISI, EI-Compendex, SCOPUS, DBLP and EBSCO, among others, and will be available in the SpringerLink Digital Library. Published poster and company papers will be indexed in EI-Compendex and EBSCO. The authors of the best selected papers will be invited to extend them for publication in edited books and in international journals indexed by ISI/JCR, SCOPUS and/or DBLP, among others, such as: - ACM Transactions on Modeling and Computer Simulation (TOMACS) - Online Information Review (OIR) - Informatics for Health and Social Care (IHSC) - Computer Science and Information Systems (ComSIS) - Telecommunication Systems Journal (TSJ) - INFORMATION - An International Interdisciplinary Journal - Journal of Organizational and End User Computing (JOEUC) - Information Researh (IR) - International Journal of Internet Protocol Technology (IJIPT) - Studies in Computational Intelligence (SCI) - Journal of Advanced Computational Intelligence and Intelligent Informatics (JACIII) - Journal of Electrical and Computer Engineering (JECE): Special Issue in Advances in Radar Technology - WSEAS Transactions on Systems (TS) - Library Review (LR) - Education for Information (EI) - International Journal of IT/Business Alignment and Governance (IJITBAG) - International Journal of Systems and Service-Oriented Engineering (IJSSOE) - International Journal of Interactive Multimedia and Artificial Intelligence (IJIMAI) IMPORTANT DATES Paper Submission: December 7, 2012 Notification of Acceptance: December 30, 2012 Camera-ready Submission: January 9, 2013 Payment of Registration, to ensure the inclusion of an accepted paper in the conference proceedings: January 11, 2013. - Kind regards, Maria Lemos WorldCIST'13 http://www.aisti.eu/worldcist13/ From lpesce at uchicago.edu Thu Nov 29 10:44:29 2012 From: lpesce at uchicago.edu (Lorenzo Pesce) Date: Thu, 29 Nov 2012 10:44:29 -0600 Subject: [Swift-user] Silly question about generating arrays Message-ID: <5017A5E5-526D-4065-8B55-33F9DD66F553@uchicago.edu> Hi -- I have a simple problem to which I would like to find a simple and elegant solution. I will use this example: int matsize[] = [5, 20, 30, 50, 100, 75, 45, 15]; file MagicSquares[] ; foreach s, i in matsize { MagicSquares[i] = magicsq (MCRPath, s); } This work fine, but let us suppose that instead I need to loop over two sets of variables: int matmax[] = [5, 20, 30, 50, 100, 75, 45, 15]; for each max, indexmax matmax { file MagicSquares[] ; matsize = "["`seq -s"," max`"]"; foreach s, i in matsize { MagicSquares[i] = magicsq (MCRPath, s); } } Of course the operation I put in bold is not going to compile and would produce a "string"(?). I can think of ways of doing it, but I realized that all methods would return an array that I am not sure is perceived as such by swift. A multidimensional arrays would probably do the trick too. What would be the "best" way of doing this? -------------- next part -------------- An HTML attachment was scrubbed... URL: From wilde at mcs.anl.gov Thu Nov 29 11:23:46 2012 From: wilde at mcs.anl.gov (Michael Wilde) Date: Thu, 29 Nov 2012 11:23:46 -0600 (CST) Subject: [Swift-user] Silly question about generating arrays In-Reply-To: <5017A5E5-526D-4065-8B55-33F9DD66F553@uchicago.edu> Message-ID: <1979720253.61136.1354209826392.JavaMail.root@zimbra.anl.gov> > I need to loop over two sets of variables: > > int matmax[] = [5, 20, 30, 50, 100, 75, 45, 15]; > > for each max, indexmax matmax { > > file MagicSquares[] prefix=@strcat("magicmat.",max,"."),suffix=".dat">; > matsize = "["`seq -s"," max`"]"; > > foreach s, i in matsize { > MagicSquares[i] = magicsq (MCRPath, s); > } > } Are you looking for a pattern like this? $ cat nestedranges.swift int matmax[] = [5, 6, 7]; foreach max, indexmax in matmax { int matsize[] = [1:max]; foreach s, i in matsize { tracef("magigquares[%i] = magicsq (MCRPath, %i)\n", max, s); } } $ swift nestedranges.swift | sort Progress: time: Thu, 29 Nov 2012 17:22:25 +0000 Final status: Thu, 29 Nov 2012 17:22:25 +0000 RunID: 20121129-1722-myxnkhg3 Swift trunk swift-r6074 cog-r3517 magigquares[5] = magicsq (MCRPath, 1) magigquares[5] = magicsq (MCRPath, 2) magigquares[5] = magicsq (MCRPath, 3) magigquares[5] = magicsq (MCRPath, 4) magigquares[5] = magicsq (MCRPath, 5) magigquares[6] = magicsq (MCRPath, 1) magigquares[6] = magicsq (MCRPath, 2) magigquares[6] = magicsq (MCRPath, 3) magigquares[6] = magicsq (MCRPath, 4) magigquares[6] = magicsq (MCRPath, 5) magigquares[6] = magicsq (MCRPath, 6) magigquares[7] = magicsq (MCRPath, 1) magigquares[7] = magicsq (MCRPath, 2) magigquares[7] = magicsq (MCRPath, 3) magigquares[7] = magicsq (MCRPath, 4) magigquares[7] = magicsq (MCRPath, 5) magigquares[7] = magicsq (MCRPath, 6) magigquares[7] = magicsq (MCRPath, 7) $ From lpesce at uchicago.edu Thu Nov 29 11:41:23 2012 From: lpesce at uchicago.edu (Lorenzo Pesce) Date: Thu, 29 Nov 2012 11:41:23 -0600 Subject: [Swift-user] Silly question about generating arrays In-Reply-To: <1979720253.61136.1354209826392.JavaMail.root@zimbra.anl.gov> References: <1979720253.61136.1354209826392.JavaMail.root@zimbra.anl.gov> Message-ID: <468D98B4-BA3E-4F41-B93C-CB9EC915F3A0@uchicago.edu> Perfect ;-) On Nov 29, 2012, at 11:23 AM, Michael Wilde wrote: >> I need to loop over two sets of variables: >> >> int matmax[] = [5, 20, 30, 50, 100, 75, 45, 15]; >> >> for each max, indexmax matmax { >> >> file MagicSquares[] > prefix=@strcat("magicmat.",max,"."),suffix=".dat">; >> matsize = "["`seq -s"," max`"]"; >> >> foreach s, i in matsize { >> MagicSquares[i] = magicsq (MCRPath, s); >> } >> } > > Are you looking for a pattern like this? > > > $ cat nestedranges.swift > > int matmax[] = [5, 6, 7]; > > foreach max, indexmax in matmax { > > int matsize[] = [1:max]; > > foreach s, i in matsize { > tracef("magigquares[%i] = magicsq (MCRPath, %i)\n", max, s); > } > } > > $ swift nestedranges.swift | sort > > Progress: time: Thu, 29 Nov 2012 17:22:25 +0000 > Final status: Thu, 29 Nov 2012 17:22:25 +0000 > > RunID: 20121129-1722-myxnkhg3 > Swift trunk swift-r6074 cog-r3517 > magigquares[5] = magicsq (MCRPath, 1) > magigquares[5] = magicsq (MCRPath, 2) > magigquares[5] = magicsq (MCRPath, 3) > magigquares[5] = magicsq (MCRPath, 4) > magigquares[5] = magicsq (MCRPath, 5) > magigquares[6] = magicsq (MCRPath, 1) > magigquares[6] = magicsq (MCRPath, 2) > magigquares[6] = magicsq (MCRPath, 3) > magigquares[6] = magicsq (MCRPath, 4) > magigquares[6] = magicsq (MCRPath, 5) > magigquares[6] = magicsq (MCRPath, 6) > magigquares[7] = magicsq (MCRPath, 1) > magigquares[7] = magicsq (MCRPath, 2) > magigquares[7] = magicsq (MCRPath, 3) > magigquares[7] = magicsq (MCRPath, 4) > magigquares[7] = magicsq (MCRPath, 5) > magigquares[7] = magicsq (MCRPath, 6) > magigquares[7] = magicsq (MCRPath, 7) > $ From nbest at ci.uchicago.edu Thu Nov 29 16:29:05 2012 From: nbest at ci.uchicago.edu (Neil Best) Date: Thu, 29 Nov 2012 16:29:05 -0600 Subject: [Swift-user] swift on midway: apps and modules Message-ID: Running on Midway, it looks like there is a problem with defining apps that come from software modules. [nbest at midway-login1 narr]$ pwd /project/joshuaelliott/narr [nbest at midway-login1 narr]$ runswift narr.swift ++ swift -config config -tc.file applist -sites.file sites.xml narr.swift Warning: Parameter grb, on line 12, shadows variable of same name on line 5 Swift trunk swift-r6083 cog-r3522 RunID: 20121129-2115-mos5x5ad Progress: time: Thu, 29 Nov 2012 21:15:40 +0000 Progress: time: Thu, 29 Nov 2012 21:15:57 +0000 Initializing:6 Progress: time: Thu, 29 Nov 2012 21:15:58 +0000 Selecting site:1022 Initializing site shared directory:1 Stage in:1 Progress: time: Thu, 29 Nov 2012 21:15:59 +0000 Selecting site:623 Stage in:399 Submitting:2 Progress: time: Thu, 29 Nov 2012 21:16:00 +0000 Selecting site:623 Stage in:388 Submitted:13 Progress: time: Thu, 29 Nov 2012 21:16:01 +0000 Selecting site:623 Stage in:371 Submitting:1 Submitted:29 Progress: time: Thu, 29 Nov 2012 21:16:02 +0000 Selecting site:623 Stage in:359 Submitting:1 Failed but can retry:41 Progress: time: Thu, 29 Nov 2012 21:16:03 +0000 Selecting site:623 Stage in:341 Active:1 Failed but can retry:59 Progress: time: Thu, 29 Nov 2012 21:16:04 +0000 Selecting site:623 Stage in:322 Submitting:1 Failed but can retry:78 Progress: time: Thu, 29 Nov 2012 21:16:05 +0000 Selecting site:623 Stage in:302 Active:1 Failed but can retry:98 . . . Progress: time: Thu, 29 Nov 2012 21:18:16 +0000 Selecting site:623 Stage in:14 Submitting:1 Failed but can retry:386 Progress: time: Thu, 29 Nov 2012 21:18:17 +0000 Selecting site:623 Stage in:10 Submitting:1 Failed but can retry:390 Progress: time: Thu, 29 Nov 2012 21:18:19 +0000 Selecting site:623 Stage in:5 Submitting:1 Failed but can retry:395 Progress: time: Thu, 29 Nov 2012 21:18:20 +0000 Selecting site:622 Stage in:2 Failed:3 Failed but can retry:397 Execution failed: Exception in cnvgrib: Arguments: [-g12, -nv, data/grb/197901/narr-a_221_19790116_0600_000.grb, data/grb2/197901/narr-a_221_19790116_0600_000.grb2] Host: cluster Directory: narr-20121129-2115-mos5x5ad/jobs/r/cnvgrib-ricszn1l Caused by: Job failed with and exit code of 127 org.globus.cog.abstraction.impl.common.execution.JobException: Job failed with and exit code of 127 (exit code: 127) at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:38) at org.globus.cog.karajan.workflow.service.handlers.RequestHandler.receiveCompleted(RequestHandler.java:90) at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.handleRequest(AbstractKarajanChannel.java:502) at org.globus.cog.karajan.workflow.service.channels.AbstractStreamKarajanChannel.stepNIO(AbstractStreamKarajanChannel.java:238) at org.globus.cog.karajan.workflow.service.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97) at org.globus.cog.karajan.workflow.service.channels.NIOMultiplexer.run(NIOMultiplexer.java:56) (exit code: 127) cnvgrib, narr.swift, line 17 [nbest at midway-login1 narr]$ cat applist cluster cnvgrib /software/cnvgrib-1.4-el6-x86_64/bin/cnvgrib null null null cluster wgrib2 /software/wgrib2-0.1-el6-x86_64/bin/wgrib2 null null null My app definition looks like this: app (file grb2) cnvgrib (file grb) { cnvgrib "-g12" "-nv" @grb @grb2; } I don't see the directory referenced in the "Exception" stanza. Where should that be? Does this have anything to do with the fact that I am calling an executable from Modules, and therefore the Swift workers have bad environments? From davidk at ci.uchicago.edu Thu Nov 29 17:04:25 2012 From: davidk at ci.uchicago.edu (David Kelly) Date: Thu, 29 Nov 2012 17:04:25 -0600 (CST) Subject: [Swift-user] swift on midway: apps and modules In-Reply-To: Message-ID: <364676584.130715.1354230265164.JavaMail.root@zimbra-mb2.anl.gov> Hi Neil, It could possibly be something environmental. Are the modules loaded automatically when you log in through your .bashrc? If they are not already, could you give that a try? If that doesn't work, try editing sites.xml and try changing /scratch/local/swift to /project/joshuaelliott/narr Then try the run again and let me know when it's finished. That will just make some extra debugging information available that might better explain why it's failing. Thanks, David ----- Original Message ----- > From: "Neil Best" > To: swift-user at ci.uchicago.edu > Sent: Thursday, November 29, 2012 4:29:05 PM > Subject: [Swift-user] swift on midway: apps and modules > Running on Midway, it looks like there is a problem with defining apps > that come from software modules. > > [nbest at midway-login1 narr]$ pwd > /project/joshuaelliott/narr > [nbest at midway-login1 narr]$ runswift narr.swift > ++ swift -config config -tc.file applist -sites.file sites.xml > narr.swift > Warning: Parameter grb, on line 12, shadows variable of same name on > line 5 > Swift trunk swift-r6083 cog-r3522 > > RunID: 20121129-2115-mos5x5ad > Progress: time: Thu, 29 Nov 2012 21:15:40 +0000 > Progress: time: Thu, 29 Nov 2012 21:15:57 +0000 Initializing:6 > Progress: time: Thu, 29 Nov 2012 21:15:58 +0000 Selecting site:1022 > Initializing site shared directory:1 Stage in:1 > Progress: time: Thu, 29 Nov 2012 21:15:59 +0000 Selecting site:623 > Stage in:399 Submitting:2 > Progress: time: Thu, 29 Nov 2012 21:16:00 +0000 Selecting site:623 > Stage in:388 Submitted:13 > Progress: time: Thu, 29 Nov 2012 21:16:01 +0000 Selecting site:623 > Stage in:371 Submitting:1 Submitted:29 > Progress: time: Thu, 29 Nov 2012 21:16:02 +0000 Selecting site:623 > Stage in:359 Submitting:1 Failed but can retry:41 > Progress: time: Thu, 29 Nov 2012 21:16:03 +0000 Selecting site:623 > Stage in:341 Active:1 Failed but can retry:59 > Progress: time: Thu, 29 Nov 2012 21:16:04 +0000 Selecting site:623 > Stage in:322 Submitting:1 Failed but can retry:78 > Progress: time: Thu, 29 Nov 2012 21:16:05 +0000 Selecting site:623 > Stage in:302 Active:1 Failed but can retry:98 > . . . > Progress: time: Thu, 29 Nov 2012 21:18:16 +0000 Selecting site:623 > Stage in:14 Submitting:1 Failed but can retry:386 > Progress: time: Thu, 29 Nov 2012 21:18:17 +0000 Selecting site:623 > Stage in:10 Submitting:1 Failed but can retry:390 > Progress: time: Thu, 29 Nov 2012 21:18:19 +0000 Selecting site:623 > Stage in:5 Submitting:1 Failed but can retry:395 > Progress: time: Thu, 29 Nov 2012 21:18:20 +0000 Selecting site:622 > Stage in:2 Failed:3 Failed but can retry:397 > Execution failed: > Exception in cnvgrib: > Arguments: [-g12, -nv, > data/grb/197901/narr-a_221_19790116_0600_000.grb, > data/grb2/197901/narr-a_221_19790116_0600_000.grb2] > Host: cluster > Directory: narr-20121129-2115-mos5x5ad/jobs/r/cnvgrib-ricszn1l > Caused by: > Job failed with and exit code of 127 > org.globus.cog.abstraction.impl.common.execution.JobException: Job > failed with and exit code of 127 (exit code: 127) > at > org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:38) > at > org.globus.cog.karajan.workflow.service.handlers.RequestHandler.receiveCompleted(RequestHandler.java:90) > at > org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.handleRequest(AbstractKarajanChannel.java:502) > at > org.globus.cog.karajan.workflow.service.channels.AbstractStreamKarajanChannel.stepNIO(AbstractStreamKarajanChannel.java:238) > at > org.globus.cog.karajan.workflow.service.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97) > at > org.globus.cog.karajan.workflow.service.channels.NIOMultiplexer.run(NIOMultiplexer.java:56) > (exit code: 127) > cnvgrib, narr.swift, line 17 > [nbest at midway-login1 narr]$ cat applist > cluster cnvgrib /software/cnvgrib-1.4-el6-x86_64/bin/cnvgrib null null > null > cluster wgrib2 /software/wgrib2-0.1-el6-x86_64/bin/wgrib2 null null > null > > My app definition looks like this: > > app (file grb2) cnvgrib (file grb) { > cnvgrib "-g12" "-nv" @grb @grb2; > } > > I don't see the directory referenced in the "Exception" stanza. Where > should that be? > > Does this have anything to do with the fact that I am calling an > executable from Modules, and therefore the Swift workers have bad > environments? > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user From nbest at ci.uchicago.edu Thu Nov 29 18:15:26 2012 From: nbest at ci.uchicago.edu (Neil Best) Date: Thu, 29 Nov 2012 18:15:26 -0600 Subject: [Swift-user] swift on midway: apps and modules In-Reply-To: <364676584.130715.1354230265164.JavaMail.root@zimbra-mb2.anl.gov> References: <364676584.130715.1354230265164.JavaMail.root@zimbra-mb2.anl.gov> Message-ID: On Thu, Nov 29, 2012 at 5:04 PM, David Kelly wrote: > It could possibly be something environmental. Are the modules loaded automatically when you log in through your .bashrc? If they are not already, could you give that a try? Good thinking. There is a .modulerc file that gets sourced through some mechanism at login other then ~/.bashrc (not sure how) so I added it there for good measure. > > If that doesn't work, try editing sites.xml and try changing > > /scratch/local/swift > to > /project/joshuaelliott/narr > > Then try the run again and let me know when it's finished. That will just make some extra debugging information available that might better explain why it's failing. > I tried that also and now got a new error, so that's progress, right? It's complaining that it can't open a file that one of the utilities needs. I presume that you can get in and look at logs so I won't take the time to include snippets here, but maybe that's a bas assumption. If we could get this going tomorrow that would be great. Take a look at /project/joshuaelliott/narr/narr-20121130-0005-4uhyfkca. Thanks. From davidk at ci.uchicago.edu Thu Nov 29 19:15:35 2012 From: davidk at ci.uchicago.edu (David Kelly) Date: Thu, 29 Nov 2012 19:15:35 -0600 (CST) Subject: [Swift-user] swift on midway: apps and modules In-Reply-To: Message-ID: <1553442767.131199.1354238135824.JavaMail.root@zimbra-mb2.anl.gov> Neil, Right, it looks like your app is complaining because it is missing a file called nc_table. I think what you need to do is something like this in your script: file nctable<"nc_table">; Then pass that file as an argument to your app call: grb2[ix] = cnvgrib(g, nctable); Swift will create a new working directory on the worker node for each task. Passing the file as an argument to the app will make sure that it gets staged in and will available to your application. Hopefully this should do the trick. ----- Original Message ----- > From: "Neil Best" > To: "David Kelly" > Cc: swift-user at ci.uchicago.edu > Sent: Thursday, November 29, 2012 6:15:26 PM > Subject: Re: [Swift-user] swift on midway: apps and modules > On Thu, Nov 29, 2012 at 5:04 PM, David Kelly > wrote: > > > It could possibly be something environmental. Are the modules loaded > > automatically when you log in through your .bashrc? If they are not > > already, could you give that a try? > > Good thinking. There is a .modulerc file that gets sourced through > some mechanism at login other then ~/.bashrc (not sure how) so I added > it there for good measure. > > > > > > If that doesn't work, try editing sites.xml and try changing > > > > /scratch/local/swift > > to > > /project/joshuaelliott/narr > > > > Then try the run again and let me know when it's finished. That will > > just make some extra debugging information available that might > > better explain why it's failing. > > > > I tried that also and now got a new error, so that's progress, right? > It's complaining that it can't open a file that one of the utilities > needs. I presume that you can get in and look at logs so I won't take > the time to include snippets here, but maybe that's a bas assumption. > If we could get this going tomorrow that would be great. Take a look > at /project/joshuaelliott/narr/narr-20121130-0005-4uhyfkca. Thanks. From nbest at ci.uchicago.edu Thu Nov 29 22:02:19 2012 From: nbest at ci.uchicago.edu (Neil Best) Date: Thu, 29 Nov 2012 22:02:19 -0600 Subject: [Swift-user] swift on midway: apps and modules In-Reply-To: <1553442767.131199.1354238135824.JavaMail.root@zimbra-mb2.anl.gov> References: <1553442767.131199.1354238135824.JavaMail.root@zimbra-mb2.anl.gov> Message-ID: On Thu, Nov 29, 2012 at 7:15 PM, David Kelly wrote: > file nctable<"nc_table">; It's cruising now, David. Thanks for the tip. From wilde at mcs.anl.gov Thu Nov 29 23:02:38 2012 From: wilde at mcs.anl.gov (Michael Wilde) Date: Thu, 29 Nov 2012 23:02:38 -0600 (CST) Subject: [Swift-user] assign temporary file to mapper In-Reply-To: Message-ID: <123370129.62424.1354251758581.JavaMail.root@zimbra.anl.gov> Hao, my apologies - I lost track of your problem. Can you send your log file from this run? it should be a file with your scriptname, then -20121120-1538-mt4hjjd6.log. Something is inconsistent here: your output shows trace lines that are not in the source code you sent. The log file will have the source code that matches the run which produced there error. I may be missing something here, but it looks like you are concatenating two mapped temporary filenames and mapping yet another filename to that value: File not found: /users/slic/tlai/work/mapReduceScripts/./_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--\ array/_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array/elt- But I dont see where this is happening the source code you sent. - Mike ----- Original Message ----- > From: "Hao Yang" > To: "Michael Wilde" > Cc: mosastore at googlegroups.com, swift-user at ci.uchicago.edu > Sent: Tuesday, November 20, 2012 10:33:15 PM > Subject: Re: [Swift-user] assign temporary file to mapper > Hi, Michael: > > > The output_chunks[2] is assigned in the iterate via output_chunks[i] = > appAddFiles(processed_chunks[i], output_chunks[i-1]). (I put the > script code and the error message below). > > > // swift code > type RSFile; // rsf file > type StringFile; // File that stores strings > > > // APPs > // getNDims: Extract last dimension from rsf file > app (StringFile ndims) appGetNDims(RSFile input) { > sffiledims_script @filename(input) "1" stdout=@filename(ndims); > } > > > // getDims: Extract dimension list from rsf file > app (StringFile dims) appGetDims(RSFile input) { > sffiledims_script @filename(input) "2" stdout=@filename(dims); > } > > > // getWindow: Window a slice from rsf file > app (RSFile out) appSplitFile(RSFile inp, string dim, int slice) { > sfwindow_script @filename(inp) @filename(out) dim slice; > } > > > // addFiles: adds inp1 and inp2 and returns out > app (RSFile out) appAddFiles(RSFile inp1, RSFile inp2) { > sfadd_script @filename(out) @filename(inp1) @filename(inp2); > } > > > // In-core variables > string input_name; // input name of the file > string output_name; // output name of the file > string last_dim; // Last dimension of file (default splitting > dimension) > string split_dim; // Splitting dimension > string dims_list; // string of dimensions (unsplitted) > string dims_array[]; // Array of string of dimensions (splitted) > int dim_length; // length of splitted dimension > > > // File variables > // Setup input and output files > RSFile input_data ; > RSFile output_data ; > StringFile ndims_file; // number of dimensions of the input file > StringFile dims_file; // dimension array of the input file > RSFile input_chunks[]; // Splitted input > RSFile processed_chunks[]; // data chunks after parallel processing > RSFile output_chunks[]; // Temporary output array for reduction > > > // main part > // Define and parse input arguments > input_name = @arg("in","in.rsf"); > output_name = @arg("out","out.rsf"); > > > // extract dims and setup variables > ndims_file = appGetNDims(input_data); > dims_file = appGetDims(input_data); > > > // Define and parse split dimension and dimensions array > last_dim = readData(ndims_file); > split_dim = @arg("dim", last_dim); > dims_list = readData(dims_file); > dims_array = @strsplit(dims_list,","); > dim_length = @toint(dims_array[@toint(split_dim)-1]); > trace("Dim length = ", dim_length); > > > // Split the files parallelly > foreach i in [0:dim_length-1] { > trace("Splitting ", i); > input_chunks[i] = appSplitFile(input_data, split_dim, i+1); > trace("Working furiously on ", @filename(input_chunks[i])); > trace("Chunk number: ", i); > processed_chunks[i] = input_chunks[i]; > } > > > // Add all the chunks together > iterate i { > if( i == 0 ) { > trace("Setting chunk 0 to ", @filename(output_chunks[0])); > output_chunks[0] = processed_chunks[0]; > } else { > trace("Adding ", i); > output_chunks[i] = appAddFiles(processed_chunks[i], > output_chunks[i-1]); > } > } until (i == dim_length-1); > > > output_data = output_chunks[dim_length-1]; > > > // swift message > > RunID: 20121120-1538-mt4hjjd6 > Progress: time: Tue, 20 Nov 2012 15:38:03 -0800 > SwiftScript trace: Setting chunk 0 to , > _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-0 > SwiftScript trace: filename of output_chunks[0] is, > _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-0 > SwiftScript trace: filename of processed_chunks[0] is, > _concurrent/processed_chunks-824d746f-2a4d-48b6-8b51-e2534f1f86ca--array//elt-0 > Progress: time: Tue, 20 Nov 2012 15:38:04 -0800 Submitting:1 Finished > successfully:1 > SwiftScript trace: Dim length = , 3.0 > SwiftScript trace: Splitting , 1.0 > > SwiftScript trace: Chunk number: , 1.0 > SwiftScript trace: Chunk number: , 2.0 > SwiftScript trace: Working furiously on , > _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-1 > > SwiftScript trace: Splitting , 0.0 > SwiftScript trace: Chunk number: , 0.0 > SwiftScript trace: Splitting , 2.0 > SwiftScript trace: Working furiously on , > _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-2 > SwiftScript trace: Working furiously on , > _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-0 > Progress: time: Tue, 20 Nov 2012 15:38:05 -0800 Selecting site:1 Stage > in:1 Finished successfully:3 > Progress: time: Tue, 20 Nov 2012 15:38:06 -0800 Checking status:1 > Finished successfully:4 > SwiftScript trace: Adding , 1.0 > SwiftScript trace: filename of output_chunks[i] is, > _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-1 > SwiftScript trace: filename of output_chunks[i-1] is, > _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-0 > SwiftScript trace: filename of processed_chunks[i] is, > _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-1 > SwiftScript trace: Adding , 2.0 > SwiftScript trace: filename of processed_chunks[i] is, > _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-2 > SwiftScript trace: filename of output_chunks[i] is, > _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-2 > SwiftScript trace: filename of output_chunks[i-1] is, > _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-1 > Execution failed: > File not found: > /users/slic/tlai/work/mapReduceScripts/./_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array/_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array/elt-2 > > > From the trace we can see filename of output_chunks[2] is > _concurrent/output_chunks- 33fce3ec-f16c-4430-b086- > cc90f611a0a2--array//elt-2 (we also verified the file is there after > the execution failure). > > > Thank you. > > On Tue, Nov 20, 2012 at 6:29 PM, Michael Wilde < wilde at mcs.anl.gov > > wrote: > > > Hao, > > Are you sure you assigned a value to output_chunks[2]? > > Can you post the entire program, and the entire error message? > > Thanks, > > - Mike > > > > ----- Original Message ----- > > From: "Hao Yang" < yanghao0614 at gmail.com > > > To: swift-user at ci.uchicago.edu > > Cc: mosastore at googlegroups.com > > Sent: Tuesday, November 20, 2012 7:22:45 PM > > Subject: [Swift-user] assign temporary file to mapper > > Hi, all: > > > > > > I have some problem assigning temporary file to some mapper variable > > (explained in the code below). > > > > > > type RSFile; > > string output_name; > > output_name = @arg("out","out.rsf"); > > RSFile output_data ; > > RSFile output_chunks[]; > > // some calculation to compute output_chunks[0], output_chunks[1], > > output_chunks[2] > > // at the end > > output_data = output_chunks[2]; > > > > > > > > When I try this script, swift reports " File output_chunks[2] not > > found " error and failed. But the output_chunks[2] is still > > available > > in the directory generated by swift ( _ concurrent/output_chunks- > > ae9bd0b3-ad88-45b0-95f4- f036d2d70e58--array). Should this error > > > message be expected? > > > > > > Thank you. > > > > -- > > Best Regards, > > Hao Yang > > > > The University of British Columbia > > > > > > > > > > > > _______________________________________________ > > Swift-user mailing list > > Swift-user at ci.uchicago.edu > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > -- > Michael Wilde > Computation Institute, University of Chicago > Mathematics and Computer Science Division > Argonne National Laboratory > > > > > > -- > Best Regards, > Hao Yang > Networked Systems Laboratory > Electrical and Computer Engineering > The University of British Columbia -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory From yanghao0614 at gmail.com Fri Nov 30 01:32:23 2012 From: yanghao0614 at gmail.com (Hao Yang) Date: Thu, 29 Nov 2012 23:32:23 -0800 Subject: [Swift-user] assign temporary file to mapper In-Reply-To: <123370129.62424.1354251758581.JavaMail.root@zimbra.anl.gov> References: <123370129.62424.1354251758581.JavaMail.root@zimbra.anl.gov> Message-ID: Hi, Michael: Please find the log file in the attachment. In the source code we declared a temporary file array *RSFile output_chunks[]*, and a single file mapper. *RSFile output_data *. Then we used the iterate at the end of the source code to get output_chunks[0], ..., till output_chunks[dim_length - 1]. The error happened when Swift tried to execute the last line *output_data = output_chunks[dim_length-1]. *It couldn't find the file output_chunks[dim_length-1], but we verified the file was available in the _concurrent directory. So we were wondering if it is valid to assign temporary file to file mapper. Thank you. On Thu, Nov 29, 2012 at 9:02 PM, Michael Wilde wrote: > Hao, my apologies - I lost track of your problem. > > Can you send your log file from this run? it should be a file with your > scriptname, then -20121120-1538-mt4hjjd6.log. > > Something is inconsistent here: your output shows trace lines that are not > in the source code you sent. The log file will have the source code that > matches the run which produced there error. > > I may be missing something here, but it looks like you are concatenating > two mapped temporary filenames and mapping yet another filename to that > value: > > File not found: > /users/slic/tlai/work/mapReduceScripts/./_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--\ > > array/_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array/elt- > > But I dont see where this is happening the source code you sent. > > - Mike > > ----- Original Message ----- > > From: "Hao Yang" > > To: "Michael Wilde" > > Cc: mosastore at googlegroups.com, swift-user at ci.uchicago.edu > > Sent: Tuesday, November 20, 2012 10:33:15 PM > > Subject: Re: [Swift-user] assign temporary file to mapper > > Hi, Michael: > > > > > > The output_chunks[2] is assigned in the iterate via output_chunks[i] = > > appAddFiles(processed_chunks[i], output_chunks[i-1]). (I put the > > script code and the error message below). > > > > > > // swift code > > type RSFile; // rsf file > > type StringFile; // File that stores strings > > > > > > // APPs > > // getNDims: Extract last dimension from rsf file > > app (StringFile ndims) appGetNDims(RSFile input) { > > sffiledims_script @filename(input) "1" stdout=@filename(ndims); > > } > > > > > > // getDims: Extract dimension list from rsf file > > app (StringFile dims) appGetDims(RSFile input) { > > sffiledims_script @filename(input) "2" stdout=@filename(dims); > > } > > > > > > // getWindow: Window a slice from rsf file > > app (RSFile out) appSplitFile(RSFile inp, string dim, int slice) { > > sfwindow_script @filename(inp) @filename(out) dim slice; > > } > > > > > > // addFiles: adds inp1 and inp2 and returns out > > app (RSFile out) appAddFiles(RSFile inp1, RSFile inp2) { > > sfadd_script @filename(out) @filename(inp1) @filename(inp2); > > } > > > > > > // In-core variables > > string input_name; // input name of the file > > string output_name; // output name of the file > > string last_dim; // Last dimension of file (default splitting > > dimension) > > string split_dim; // Splitting dimension > > string dims_list; // string of dimensions (unsplitted) > > string dims_array[]; // Array of string of dimensions (splitted) > > int dim_length; // length of splitted dimension > > > > > > // File variables > > // Setup input and output files > > RSFile input_data ; > > RSFile output_data ; > > StringFile ndims_file; // number of dimensions of the input file > > StringFile dims_file; // dimension array of the input file > > RSFile input_chunks[]; // Splitted input > > RSFile processed_chunks[]; // data chunks after parallel processing > > RSFile output_chunks[]; // Temporary output array for reduction > > > > > > // main part > > // Define and parse input arguments > > input_name = @arg("in","in.rsf"); > > output_name = @arg("out","out.rsf"); > > > > > > // extract dims and setup variables > > ndims_file = appGetNDims(input_data); > > dims_file = appGetDims(input_data); > > > > > > // Define and parse split dimension and dimensions array > > last_dim = readData(ndims_file); > > split_dim = @arg("dim", last_dim); > > dims_list = readData(dims_file); > > dims_array = @strsplit(dims_list,","); > > dim_length = @toint(dims_array[@toint(split_dim)-1]); > > trace("Dim length = ", dim_length); > > > > > > // Split the files parallelly > > foreach i in [0:dim_length-1] { > > trace("Splitting ", i); > > input_chunks[i] = appSplitFile(input_data, split_dim, i+1); > > trace("Working furiously on ", @filename(input_chunks[i])); > > trace("Chunk number: ", i); > > processed_chunks[i] = input_chunks[i]; > > } > > > > > > // Add all the chunks together > > iterate i { > > if( i == 0 ) { > > trace("Setting chunk 0 to ", @filename(output_chunks[0])); > > output_chunks[0] = processed_chunks[0]; > > } else { > > trace("Adding ", i); > > output_chunks[i] = appAddFiles(processed_chunks[i], > > output_chunks[i-1]); > > } > > } until (i == dim_length-1); > > > > > > output_data = output_chunks[dim_length-1]; > > > > > > // swift message > > > > RunID: 20121120-1538-mt4hjjd6 > > Progress: time: Tue, 20 Nov 2012 15:38:03 -0800 > > SwiftScript trace: Setting chunk 0 to , > > > _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-0 > > SwiftScript trace: filename of output_chunks[0] is, > > > _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-0 > > SwiftScript trace: filename of processed_chunks[0] is, > > > _concurrent/processed_chunks-824d746f-2a4d-48b6-8b51-e2534f1f86ca--array//elt-0 > > Progress: time: Tue, 20 Nov 2012 15:38:04 -0800 Submitting:1 Finished > > successfully:1 > > SwiftScript trace: Dim length = , 3.0 > > SwiftScript trace: Splitting , 1.0 > > > > SwiftScript trace: Chunk number: , 1.0 > > SwiftScript trace: Chunk number: , 2.0 > > SwiftScript trace: Working furiously on , > > > _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-1 > > > > SwiftScript trace: Splitting , 0.0 > > SwiftScript trace: Chunk number: , 0.0 > > SwiftScript trace: Splitting , 2.0 > > SwiftScript trace: Working furiously on , > > > _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-2 > > SwiftScript trace: Working furiously on , > > > _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-0 > > Progress: time: Tue, 20 Nov 2012 15:38:05 -0800 Selecting site:1 Stage > > in:1 Finished successfully:3 > > Progress: time: Tue, 20 Nov 2012 15:38:06 -0800 Checking status:1 > > Finished successfully:4 > > SwiftScript trace: Adding , 1.0 > > SwiftScript trace: filename of output_chunks[i] is, > > > _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-1 > > SwiftScript trace: filename of output_chunks[i-1] is, > > > _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-0 > > SwiftScript trace: filename of processed_chunks[i] is, > > > _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-1 > > SwiftScript trace: Adding , 2.0 > > SwiftScript trace: filename of processed_chunks[i] is, > > > _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-2 > > SwiftScript trace: filename of output_chunks[i] is, > > > _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-2 > > SwiftScript trace: filename of output_chunks[i-1] is, > > > _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-1 > > Execution failed: > > File not found: > > > /users/slic/tlai/work/mapReduceScripts/./_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array/_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array/elt-2 > > > > > > From the trace we can see filename of output_chunks[2] is > > _concurrent/output_chunks- 33fce3ec-f16c-4430-b086- > > cc90f611a0a2--array//elt-2 (we also verified the file is there after > > the execution failure). > > > > > > Thank you. > > > > On Tue, Nov 20, 2012 at 6:29 PM, Michael Wilde < wilde at mcs.anl.gov > > > wrote: > > > > > > Hao, > > > > Are you sure you assigned a value to output_chunks[2]? > > > > Can you post the entire program, and the entire error message? > > > > Thanks, > > > > - Mike > > > > > > > > ----- Original Message ----- > > > From: "Hao Yang" < yanghao0614 at gmail.com > > > > To: swift-user at ci.uchicago.edu > > > Cc: mosastore at googlegroups.com > > > Sent: Tuesday, November 20, 2012 7:22:45 PM > > > Subject: [Swift-user] assign temporary file to mapper > > > Hi, all: > > > > > > > > > I have some problem assigning temporary file to some mapper variable > > > (explained in the code below). > > > > > > > > > type RSFile; > > > string output_name; > > > output_name = @arg("out","out.rsf"); > > > RSFile output_data ; > > > RSFile output_chunks[]; > > > // some calculation to compute output_chunks[0], output_chunks[1], > > > output_chunks[2] > > > // at the end > > > output_data = output_chunks[2]; > > > > > > > > > > > > When I try this script, swift reports " File output_chunks[2] not > > > found " error and failed. But the output_chunks[2] is still > > > available > > > in the directory generated by swift ( _ concurrent/output_chunks- > > > ae9bd0b3-ad88-45b0-95f4- f036d2d70e58--array). Should this error > > > > > message be expected? > > > > > > > > > Thank you. > > > > > > -- > > > Best Regards, > > > Hao Yang > > > > > > The University of British Columbia > > > > > > > > > > > > > > > > > > _______________________________________________ > > > Swift-user mailing list > > > Swift-user at ci.uchicago.edu > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > > > -- > > Michael Wilde > > Computation Institute, University of Chicago > > Mathematics and Computer Science Division > > Argonne National Laboratory > > > > > > > > > > > > -- > > Best Regards, > > Hao Yang > > Networked Systems Laboratory > > Electrical and Computer Engineering > > The University of British Columbia > > -- > Michael Wilde > Computation Institute, University of Chicago > Mathematics and Computer Science Division > Argonne National Laboratory > > -- Best Regards, Hao Yang Networked Systems Laboratory Electrical and Computer Engineering The University of British Columbia -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: bleh.log Type: application/octet-stream Size: 55164 bytes Desc: not available URL: From ketancmaheshwari at gmail.com Fri Nov 30 09:48:47 2012 From: ketancmaheshwari at gmail.com (Ketan Maheshwari) Date: Fri, 30 Nov 2012 10:48:47 -0500 Subject: [Swift-user] run exe from the same dir as staged data Message-ID: Hi, I am trying to swiftify an application whose executable expects the input data (4 files) to be in the directory from where the call is made. The said executable (vasp), looks for these files in the pwd and runs if the files are found. However, using swift, the files are staged with dirtree and the call is made via _swiftwrap from /job/x/shared which is causing the application failure. I could manually cd into the staged dirtree where the inputs are present and make a call to vasp and it works. Same behavior is observed with swift trunk, and 0.93. Is there a workaround for the above pattern? Attached are the logs for one sample run. Thanks, -- Ketan -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: vaspswiftlog.tgz Type: application/x-gzip Size: 110544 bytes Desc: not available URL: From wilde at mcs.anl.gov Fri Nov 30 11:00:54 2012 From: wilde at mcs.anl.gov (Michael Wilde) Date: Fri, 30 Nov 2012 11:00:54 -0600 (CST) Subject: [Swift-user] run exe from the same dir as staged data In-Reply-To: Message-ID: <935895546.63130.1354294854305.JavaMail.root@zimbra.anl.gov> Hi Ketan, > I am trying to swiftify an application whose executable expects the > input data (4 files) to be in the directory from where the call is > made. The said executable (vasp), looks for these files in the pwd and > runs if the files are found. The 4 files are symlinked by swift from the workdir shared/ directory into the current working directory (which swift calls the "job directory"). This is explained in the Swift User Guide: http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html#_how_swift_implements_the_site_execution_model However, in your swift script you mapped the input files to fully qualified pathnames, so they show up under your shared/ directory: $ pwd /Users/wilde/Downloads/vaspswiftlog/swiftwork/vasp-20121130-1006-v9x90821 $ find * -type f 3 info/y/vasp-yt8l7p1l-info shared/_swiftseq shared/_swiftwrap shared/home/kcm92/vasp-work/fullpipeline_small_scale/encut_variations/300/INCAR shared/home/kcm92/vasp-work/fullpipeline_small_scale/encut_variations/300/KPOINTS shared/home/kcm92/vasp-work/fullpipeline_small_scale/encut_variations/300/POSCAR shared/home/kcm92/vasp-work/fullpipeline_small_scale/encut_variations/300/POTCAR shared/stdouts/f.0000.out wrapper.log $ These will be linked below your job dir as, e.g.: ./home/kcm92/vasp-work/fullpipeline_small_scale/encut_variations/300/INCAR but it seems like vasp is expecting ./INCAR instead. One solution is: - run vasp from a wrapper script, like "runvasp.sh" - place one of the files, say the INCAR file, on the command line in the wrapper's Swift app function. - in the wrapper, do: cd $(dirname $1) - make sure the wrapper creates its output files where swift is expecting, ie either in $(dirname $1) or all the way back up in the jobdir, as you have it now. The easiest way to do this is to also pass the names of the stdout file as an arg to the wrapper (eg $2) - you should always capture and return stderr as well, if thats an affordable overhead (ie if you dont have too many files). Also, I see in your stdout file the following: $ cat shared/stdouts/f.0000.out Error reading item 'IMAGES' from file INCAR. CMA: unable to get RDMA device list That CMA message suggests that your version of VASP may be trying to execute under MPI. You should test VASP on a login host or manually on a cluster node to verify that it will run without MPI. Or, we can help you run it under MPI as well; the Swift MPI documentation needs to be revised for the latest trunk code / 0.94 release candidate. Mike > However, using swift, the files are staged with dirtree and the call > is made via _swiftwrap from /job/x/shared which is causing > the application failure. I could manually cd into the staged dirtree > where the inputs are present and make a call to vasp and it works. > > > Same behavior is observed with swift trunk, and 0.93. > > > Is there a workaround for the above pattern? > > > Attached are the logs for one sample run. > > > > Thanks, -- > Ketan > > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory From nbest at ci.uchicago.edu Fri Nov 30 13:08:55 2012 From: nbest at ci.uchicago.edu (Neil Best) Date: Fri, 30 Nov 2012 13:08:55 -0600 Subject: [Swift-user] swift on midway: apps and modules In-Reply-To: References: <1553442767.131199.1354238135824.JavaMail.root@zimbra-mb2.anl.gov> Message-ID: It seems like everything is slowing down: [nbest at midway-login1 narr]$ date;find data/grb2 -type f | wc -l Thu Nov 29 21:30:34 CST 2012 8061 [nbest at midway-login1 narr]$ date;find data/grb2 -type f | wc -l Thu Nov 29 21:48:01 CST 2012 12680 [nbest at midway-login1 narr]$ date;find data/grb2 -type f | wc -l Thu Nov 29 22:01:36 CST 2012 16288 [nbest at midway-login1 narr]$ date;find data/grb2 -type f | wc -l Fri Nov 30 10:38:56 CST 2012 19238 [nbest at midway-login1 narr]$ date;find data/grb2 -type f | wc -l Fri Nov 30 12:58:04 CST 2012 19238 [nbest at midway-login1 narr]$ date;find data/nc -type f | wc -l Fri Nov 30 12:58:45 CST 2012 18419 [nbest at midway-login1 narr]$ pwd /project/joshuaelliott/narr Too bad I didn't watch the nc/ directory as well, but both numbers should be climbing to ~98k. The swift process on the head node seems to have stalled: Progress: time: Fri, 30 Nov 2012 04:24:02 +0000 Stage in:448 Submitted:9 Stage out:1206 Finished successfully:37070 Failed but can retry:1 Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Progress ticker" Exception in thread "PullThread" java.lang.OutOfMemoryError: Java heap space Exception in thread "Timer-1" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Scheduler" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-1" Exception in thread "pool-1-thread-41" java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space Progress: time: Fri, 30 Nov 2012 04:25:03 +0000 Stage in:447 Submitted:9 Stage out:1206 Finished successfully:37070 Failed but can retry:2 Progress: time: Fri, 30 Nov 2012 04:25:05 +0000 Stage in:446 Submitting:1 Submitted:9 Stage out:1206 Finished successfully:37070 Failed but can retry :2 Progress: time: Fri, 30 Nov 2012 04:25:07 +0000 Stage in:445 Submitting:2 Submitted:9 Stage out:1206 Finished successfully:37070 Failed but can retry :2 Progress: time: Fri, 30 Nov 2012 04:25:09 +0000 Stage in:444 Submitting:3 Submitted:9 Stage out:1206 Finished successfully:37070 Failed but can retry :2 No output since then. Do I need to restart it? What do you think is happening? Scrolling back in the output I also see this: Progress: time: Fri, 30 Nov 2012 04:19:15 +0000 Initializing:1 Stage in:446 Submitting:1 Stage out:1213 Finished successfully:37063 Failed but can re try:1 java.lang.OutOfMemoryError: Java heap space Dumping heap to java_pid25355.hprof ... Heap dump file created [1172799498 bytes in 10.727 secs] Progress: time: Fri, 30 Nov 2012 04:19:35 +0000Exception in thread "PBS provider queue poller" Initializing:1 Stage in:446 Submitting:1 Stage out:1213 Finished successfully:37063 Failed but can retry:1 Progress: time: Fri, 30 Nov 2012 04:19:41 +0000 Initializing:1 Stage in:446 Submitted:1 Stage out:1213 Finished successfully:37063 Failed but can ret ry:1 java.lang.OutOfMemoryError: Java heap space at java.io.BufferedReader.(BufferedReader.java:98) at java.io.BufferedReader.(BufferedReader.java:109) at org.globus.cog.abstraction.impl.scheduler.slurm.QueuePoller.processStdout(QueuePoller.java:75) . . . Exception in thread "Timer-2" Exception in thread "Overloaded Host Monitor" java.lang.OutOfMemoryError: Java heap space Progress: time: Fri, 30 Nov 2012 04:20:19 +0000 Initializing:1 Selecting site:1 Stage in:446 Submitted:1 Stage out:1213 Finished successfully:37063 Failed but can retry:1 Fri, 30 Nov 2012 04:20:42 +0000 Initializing:1 Selecting site:1 Stage in:446 Submitted:1 Stage out:1213 Finished successfully:37063 Failed but can re try:1 java.lang.OutOfMemoryError: Java heap space at java.util.HashMap.newValueIterator(HashMap.java:856) at java.util.HashMap$Values.iterator(HashMap.java:923) at java.util.AbstractCollection.toArray(AbstractCollection.java:137) at java.util.ArrayList.addAll(ArrayList.java:530) at org.globus.cog.karajan.workflow.service.channels.ChannelContext.getActiveCommands(ChannelContext.java:171) at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.checkTimeouts(AbstractKarajanChannel.java:124) at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel$1.run(AbstractKarajanChannel.java:118) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) Ominous. If I am restarting are there special steps that I need to take to avoid redoing work already done? Originally I thought the slow-down was due to fair-share or increased activity on the cluster. Even though I am not so sure about those theories now, can you point me to a primer on how to monitor the cluster? I am new to Slurm. So far I have found the RCC documents to only cover the broadest generalities but maybe I have overlooked something. Thanks. On Thu, Nov 29, 2012 at 10:02 PM, Neil Best wrote: > On Thu, Nov 29, 2012 at 7:15 PM, David Kelly wrote: >> file nctable<"nc_table">; > > It's cruising now, David. Thanks for the tip. From wilde at mcs.anl.gov Fri Nov 30 13:44:59 2012 From: wilde at mcs.anl.gov (Michael Wilde) Date: Fri, 30 Nov 2012 13:44:59 -0600 (CST) Subject: [Swift-user] swift on midway: apps and modules In-Reply-To: Message-ID: <375623837.63474.1354304699142.JavaMail.root@zimbra.anl.gov> > From: "Neil Best" > Sent: Friday, November 30, 2012 1:08:55 PM > It seems like everything is slowing down: > The swift process on the head node seems to have stalled: > ... > Exception: java.lang.OutOfMemoryError thrown from the > UncaughtExceptionHandler in thread "Progress ticker" > Exception in thread "PullThread" java.lang.OutOfMemoryError: Java heap > space > No output since then. Do I need to restart it? What do you think is > happening? I think that the swift command (ie the Java JVM that it runs) is out of memory. You can give it more like this: export SWIFT_HEAP_MAX=4096M # 4GB swift -config etc etc I will file a ticket to get this in the User Guide. You might be able to reduce the memory usage by lowering the value of this Swift property to something not much greater than the number of concurrent cores you expect to get from the cluster: foreach.max.threads=1024 # default If you only expect say 20 nodes x 12 cores = 240 concurrent app() calls, reduce this setting to something like 400 or 500. > If I am restarting are there special steps that I need to take to > avoid redoing work already done? In the directory in which you ran Swift you should see a file ending in .rlog with the run-id of your latest run. Re-issue the same swift command as you did to start the failing run, but add the argument: -resume.file=runid.rlog This is described in the User Guide: http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html#_restarts If you are using the runswift command from our latest RCC tutorial, you'll need to edit that script to add both the heap variable and the resume flag. > Originally I thought the slow-down was due to fair-share or increased > activity on the cluster. Even though I am not so sure about those > theories now, can you point me to a primer on how to monitor the > cluster? I am new to Slurm. So far I have found the RCC documents to > only cover the broadest generalities but maybe I have overlooked > something. Thanks. This slurm command will show what you have queued and running: squeue -u $USER -l sinfo -l # will show the queues, called "partitions" in SLURM. I sometimes do this to watch my jobs in a separate screen window: watch -n 60 squeue -u $USER -l - Mike From wilde at mcs.anl.gov Fri Nov 30 17:21:22 2012 From: wilde at mcs.anl.gov (Michael Wilde) Date: Fri, 30 Nov 2012 17:21:22 -0600 (CST) Subject: [Swift-user] assign temporary file to mapper In-Reply-To: Message-ID: <1276190277.63936.1354317682846.JavaMail.root@zimbra.anl.gov> Dear Hao, Mihael has diagnosed this problem as likely being caused by a Swift bug in 0.93 that was fixed many months ago in trunk. Can you repeat your test with the current 0.94 release candidate? It at: http://www.ci.uchicago.edu/swift/packages/swift-0.94RC1.tar.gz Thanks, and sorry again for the delay in debugging this. - Mike ----- Original Message ----- > From: "Hao Yang" > To: "Michael Wilde" > Cc: mosastore at googlegroups.com, swift-user at ci.uchicago.edu > Sent: Friday, November 30, 2012 1:32:23 AM > Subject: Re: [Swift-user] assign temporary file to mapper > Hi, Michael: > > > Please find the log file in the attachment. > > In the source code we declared a temporary file array RSFile > output_chunks[] , and a single file mapper. RSFile output_data > . Then we used the iterate at > the end of the source code to get output_chunks[0], ..., till > output_chunks[dim_length - 1]. The error happened when Swift tried to > execute the last line output_data = output_chunks[dim_length-1]. It > couldn't find the file output_chunks[dim_length-1], but we verified > the file was available in the _concurrent directory. So we were > wondering if it is valid to assign temporary file to file mapper. > > > Thank you. > > > On Thu, Nov 29, 2012 at 9:02 PM, Michael Wilde < wilde at mcs.anl.gov > > wrote: > > > Hao, my apologies - I lost track of your problem. > > Can you send your log file from this run? it should be a file with > your scriptname, then -20121120-1538-mt4hjjd6.log. > > Something is inconsistent here: your output shows trace lines that are > not in the source code you sent. The log file will have the source > code that matches the run which produced there error. > > I may be missing something here, but it looks like you are > concatenating two mapped temporary filenames and mapping yet another > filename to that value: > > File not found: > /users/slic/tlai/work/mapReduceScripts/./_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--\ > array/_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array/elt- > > But I dont see where this is happening the source code you sent. > > > - Mike > > ----- Original Message ----- > > From: "Hao Yang" < yanghao0614 at gmail.com > > > > > To: "Michael Wilde" < wilde at mcs.anl.gov > > > Cc: mosastore at googlegroups.com , swift-user at ci.uchicago.edu > > Sent: Tuesday, November 20, 2012 10:33:15 PM > > Subject: Re: [Swift-user] assign temporary file to mapper > > Hi, Michael: > > > > > > The output_chunks[2] is assigned in the iterate via output_chunks[i] > > = > > appAddFiles(processed_chunks[i], output_chunks[i-1]). (I put the > > script code and the error message below). > > > > > > // swift code > > type RSFile; // rsf file > > type StringFile; // File that stores strings > > > > > > // APPs > > // getNDims: Extract last dimension from rsf file > > app (StringFile ndims) appGetNDims(RSFile input) { > > sffiledims_script @filename(input) "1" stdout=@filename(ndims); > > } > > > > > > // getDims: Extract dimension list from rsf file > > app (StringFile dims) appGetDims(RSFile input) { > > sffiledims_script @filename(input) "2" stdout=@filename(dims); > > } > > > > > > // getWindow: Window a slice from rsf file > > app (RSFile out) appSplitFile(RSFile inp, string dim, int slice) { > > sfwindow_script @filename(inp) @filename(out) dim slice; > > } > > > > > > // addFiles: adds inp1 and inp2 and returns out > > app (RSFile out) appAddFiles(RSFile inp1, RSFile inp2) { > > sfadd_script @filename(out) @filename(inp1) @filename(inp2); > > } > > > > > > // In-core variables > > string input_name; // input name of the file > > string output_name; // output name of the file > > string last_dim; // Last dimension of file (default splitting > > dimension) > > string split_dim; // Splitting dimension > > string dims_list; // string of dimensions (unsplitted) > > string dims_array[]; // Array of string of dimensions (splitted) > > int dim_length; // length of splitted dimension > > > > > > // File variables > > // Setup input and output files > > RSFile input_data ; > > RSFile output_data ; > > StringFile ndims_file; // number of dimensions of the input file > > StringFile dims_file; // dimension array of the input file > > RSFile input_chunks[]; // Splitted input > > RSFile processed_chunks[]; // data chunks after parallel processing > > RSFile output_chunks[]; // Temporary output array for reduction > > > > > > // main part > > // Define and parse input arguments > > input_name = @arg("in","in.rsf"); > > output_name = @arg("out","out.rsf"); > > > > > > // extract dims and setup variables > > ndims_file = appGetNDims(input_data); > > dims_file = appGetDims(input_data); > > > > > > // Define and parse split dimension and dimensions array > > last_dim = readData(ndims_file); > > split_dim = @arg("dim", last_dim); > > dims_list = readData(dims_file); > > dims_array = @strsplit(dims_list,","); > > dim_length = @toint(dims_array[@toint(split_dim)-1]); > > trace("Dim length = ", dim_length); > > > > > > // Split the files parallelly > > foreach i in [0:dim_length-1] { > > trace("Splitting ", i); > > input_chunks[i] = appSplitFile(input_data, split_dim, i+1); > > trace("Working furiously on ", @filename(input_chunks[i])); > > trace("Chunk number: ", i); > > processed_chunks[i] = input_chunks[i]; > > } > > > > > > // Add all the chunks together > > iterate i { > > if( i == 0 ) { > > trace("Setting chunk 0 to ", @filename(output_chunks[0])); > > output_chunks[0] = processed_chunks[0]; > > } else { > > trace("Adding ", i); > > output_chunks[i] = appAddFiles(processed_chunks[i], > > output_chunks[i-1]); > > } > > } until (i == dim_length-1); > > > > > > output_data = output_chunks[dim_length-1]; > > > > > > // swift message > > > > RunID: 20121120-1538-mt4hjjd6 > > Progress: time: Tue, 20 Nov 2012 15:38:03 -0800 > > SwiftScript trace: Setting chunk 0 to , > > _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-0 > > SwiftScript trace: filename of output_chunks[0] is, > > _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-0 > > SwiftScript trace: filename of processed_chunks[0] is, > > _concurrent/processed_chunks-824d746f-2a4d-48b6-8b51-e2534f1f86ca--array//elt-0 > > Progress: time: Tue, 20 Nov 2012 15:38:04 -0800 Submitting:1 > > Finished > > successfully:1 > > SwiftScript trace: Dim length = , 3.0 > > SwiftScript trace: Splitting , 1.0 > > > > SwiftScript trace: Chunk number: , 1.0 > > SwiftScript trace: Chunk number: , 2.0 > > SwiftScript trace: Working furiously on , > > _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-1 > > > > SwiftScript trace: Splitting , 0.0 > > SwiftScript trace: Chunk number: , 0.0 > > SwiftScript trace: Splitting , 2.0 > > SwiftScript trace: Working furiously on , > > _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-2 > > SwiftScript trace: Working furiously on , > > _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-0 > > Progress: time: Tue, 20 Nov 2012 15:38:05 -0800 Selecting site:1 > > Stage > > in:1 Finished successfully:3 > > Progress: time: Tue, 20 Nov 2012 15:38:06 -0800 Checking status:1 > > Finished successfully:4 > > SwiftScript trace: Adding , 1.0 > > SwiftScript trace: filename of output_chunks[i] is, > > _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-1 > > SwiftScript trace: filename of output_chunks[i-1] is, > > _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-0 > > SwiftScript trace: filename of processed_chunks[i] is, > > _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-1 > > SwiftScript trace: Adding , 2.0 > > SwiftScript trace: filename of processed_chunks[i] is, > > _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-2 > > SwiftScript trace: filename of output_chunks[i] is, > > _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-2 > > SwiftScript trace: filename of output_chunks[i-1] is, > > _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-1 > > Execution failed: > > File not found: > > /users/slic/tlai/work/mapReduceScripts/./_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array/_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array/elt-2 > > > > > > From the trace we can see filename of output_chunks[2] is > > _concurrent/output_chunks- 33fce3ec-f16c-4430-b086- > > cc90f611a0a2--array//elt-2 (we also verified the file is there after > > the execution failure). > > > > > > Thank you. > > > > On Tue, Nov 20, 2012 at 6:29 PM, Michael Wilde < wilde at mcs.anl.gov > > > wrote: > > > > > > Hao, > > > > Are you sure you assigned a value to output_chunks[2]? > > > > Can you post the entire program, and the entire error message? > > > > Thanks, > > > > - Mike > > > > > > > > ----- Original Message ----- > > > From: "Hao Yang" < yanghao0614 at gmail.com > > > > To: swift-user at ci.uchicago.edu > > > Cc: mosastore at googlegroups.com > > > Sent: Tuesday, November 20, 2012 7:22:45 PM > > > Subject: [Swift-user] assign temporary file to mapper > > > Hi, all: > > > > > > > > > I have some problem assigning temporary file to some mapper > > > variable > > > (explained in the code below). > > > > > > > > > type RSFile; > > > string output_name; > > > output_name = @arg("out","out.rsf"); > > > RSFile output_data ; > > > RSFile output_chunks[]; > > > // some calculation to compute output_chunks[0], output_chunks[1], > > > output_chunks[2] > > > // at the end > > > output_data = output_chunks[2]; > > > > > > > > > > > > When I try this script, swift reports " File output_chunks[2] not > > > found " error and failed. But the output_chunks[2] is still > > > available > > > in the directory generated by swift ( _ concurrent/output_chunks- > > > ae9bd0b3-ad88-45b0-95f4- f036d2d70e58--array). Should this error > > > > > message be expected? > > > > > > > > > Thank you. > > > > > > -- > > > Best Regards, > > > Hao Yang > > > > > > The University of British Columbia > > > > > > > > > > > > > > > > > > _______________________________________________ > > > Swift-user mailing list > > > Swift-user at ci.uchicago.edu > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > > > -- > > Michael Wilde > > Computation Institute, University of Chicago > > Mathematics and Computer Science Division > > Argonne National Laboratory > > > > > > > > > > > > -- > > Best Regards, > > Hao Yang > > Networked Systems Laboratory > > Electrical and Computer Engineering > > The University of British Columbia > > -- > Michael Wilde > Computation Institute, University of Chicago > Mathematics and Computer Science Division > Argonne National Laboratory > > > > > > -- > Best Regards, > Hao Yang > Networked Systems Laboratory > Electrical and Computer Engineering > The University of British Columbia -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory From yanghao0614 at gmail.com Fri Nov 30 17:46:26 2012 From: yanghao0614 at gmail.com (Hao Yang) Date: Fri, 30 Nov 2012 15:46:26 -0800 Subject: [Swift-user] assign temporary file to mapper In-Reply-To: <1276190277.63936.1354317682846.JavaMail.root@zimbra.anl.gov> References: <1276190277.63936.1354317682846.JavaMail.root@zimbra.anl.gov> Message-ID: Hi, Michael: Thank you and Mihael for your help. We will try 0.94. On Fri, Nov 30, 2012 at 3:21 PM, Michael Wilde wrote: > Dear Hao, > > Mihael has diagnosed this problem as likely being caused by a Swift bug in > 0.93 that was fixed many months ago in trunk. Can you repeat your test > with the current 0.94 release candidate? It at: > > http://www.ci.uchicago.edu/swift/packages/swift-0.94RC1.tar.gz > > Thanks, and sorry again for the delay in debugging this. > > - Mike > > ----- Original Message ----- > > From: "Hao Yang" > > To: "Michael Wilde" > > Cc: mosastore at googlegroups.com, swift-user at ci.uchicago.edu > > Sent: Friday, November 30, 2012 1:32:23 AM > > Subject: Re: [Swift-user] assign temporary file to mapper > > Hi, Michael: > > > > > > Please find the log file in the attachment. > > > > In the source code we declared a temporary file array RSFile > > output_chunks[] , and a single file mapper. RSFile output_data > > . Then we used the iterate at > > the end of the source code to get output_chunks[0], ..., till > > output_chunks[dim_length - 1]. The error happened when Swift tried to > > execute the last line output_data = output_chunks[dim_length-1]. It > > couldn't find the file output_chunks[dim_length-1], but we verified > > the file was available in the _concurrent directory. So we were > > wondering if it is valid to assign temporary file to file mapper. > > > > > > Thank you. > > > > > > On Thu, Nov 29, 2012 at 9:02 PM, Michael Wilde < wilde at mcs.anl.gov > > > wrote: > > > > > > Hao, my apologies - I lost track of your problem. > > > > Can you send your log file from this run? it should be a file with > > your scriptname, then -20121120-1538-mt4hjjd6.log. > > > > Something is inconsistent here: your output shows trace lines that are > > not in the source code you sent. The log file will have the source > > code that matches the run which produced there error. > > > > I may be missing something here, but it looks like you are > > concatenating two mapped temporary filenames and mapping yet another > > filename to that value: > > > > File not found: > > > /users/slic/tlai/work/mapReduceScripts/./_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--\ > > > array/_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array/elt- > > > > But I dont see where this is happening the source code you sent. > > > > > > - Mike > > > > ----- Original Message ----- > > > From: "Hao Yang" < yanghao0614 at gmail.com > > > > > > > > To: "Michael Wilde" < wilde at mcs.anl.gov > > > > Cc: mosastore at googlegroups.com , swift-user at ci.uchicago.edu > > > Sent: Tuesday, November 20, 2012 10:33:15 PM > > > Subject: Re: [Swift-user] assign temporary file to mapper > > > Hi, Michael: > > > > > > > > > The output_chunks[2] is assigned in the iterate via output_chunks[i] > > > = > > > appAddFiles(processed_chunks[i], output_chunks[i-1]). (I put the > > > script code and the error message below). > > > > > > > > > // swift code > > > type RSFile; // rsf file > > > type StringFile; // File that stores strings > > > > > > > > > // APPs > > > // getNDims: Extract last dimension from rsf file > > > app (StringFile ndims) appGetNDims(RSFile input) { > > > sffiledims_script @filename(input) "1" stdout=@filename(ndims); > > > } > > > > > > > > > // getDims: Extract dimension list from rsf file > > > app (StringFile dims) appGetDims(RSFile input) { > > > sffiledims_script @filename(input) "2" stdout=@filename(dims); > > > } > > > > > > > > > // getWindow: Window a slice from rsf file > > > app (RSFile out) appSplitFile(RSFile inp, string dim, int slice) { > > > sfwindow_script @filename(inp) @filename(out) dim slice; > > > } > > > > > > > > > // addFiles: adds inp1 and inp2 and returns out > > > app (RSFile out) appAddFiles(RSFile inp1, RSFile inp2) { > > > sfadd_script @filename(out) @filename(inp1) @filename(inp2); > > > } > > > > > > > > > // In-core variables > > > string input_name; // input name of the file > > > string output_name; // output name of the file > > > string last_dim; // Last dimension of file (default splitting > > > dimension) > > > string split_dim; // Splitting dimension > > > string dims_list; // string of dimensions (unsplitted) > > > string dims_array[]; // Array of string of dimensions (splitted) > > > int dim_length; // length of splitted dimension > > > > > > > > > // File variables > > > // Setup input and output files > > > RSFile input_data ; > > > RSFile output_data ; > > > StringFile ndims_file; // number of dimensions of the input file > > > StringFile dims_file; // dimension array of the input file > > > RSFile input_chunks[]; // Splitted input > > > RSFile processed_chunks[]; // data chunks after parallel processing > > > RSFile output_chunks[]; // Temporary output array for reduction > > > > > > > > > // main part > > > // Define and parse input arguments > > > input_name = @arg("in","in.rsf"); > > > output_name = @arg("out","out.rsf"); > > > > > > > > > // extract dims and setup variables > > > ndims_file = appGetNDims(input_data); > > > dims_file = appGetDims(input_data); > > > > > > > > > // Define and parse split dimension and dimensions array > > > last_dim = readData(ndims_file); > > > split_dim = @arg("dim", last_dim); > > > dims_list = readData(dims_file); > > > dims_array = @strsplit(dims_list,","); > > > dim_length = @toint(dims_array[@toint(split_dim)-1]); > > > trace("Dim length = ", dim_length); > > > > > > > > > // Split the files parallelly > > > foreach i in [0:dim_length-1] { > > > trace("Splitting ", i); > > > input_chunks[i] = appSplitFile(input_data, split_dim, i+1); > > > trace("Working furiously on ", @filename(input_chunks[i])); > > > trace("Chunk number: ", i); > > > processed_chunks[i] = input_chunks[i]; > > > } > > > > > > > > > // Add all the chunks together > > > iterate i { > > > if( i == 0 ) { > > > trace("Setting chunk 0 to ", @filename(output_chunks[0])); > > > output_chunks[0] = processed_chunks[0]; > > > } else { > > > trace("Adding ", i); > > > output_chunks[i] = appAddFiles(processed_chunks[i], > > > output_chunks[i-1]); > > > } > > > } until (i == dim_length-1); > > > > > > > > > output_data = output_chunks[dim_length-1]; > > > > > > > > > // swift message > > > > > > RunID: 20121120-1538-mt4hjjd6 > > > Progress: time: Tue, 20 Nov 2012 15:38:03 -0800 > > > SwiftScript trace: Setting chunk 0 to , > > > > _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-0 > > > SwiftScript trace: filename of output_chunks[0] is, > > > > _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-0 > > > SwiftScript trace: filename of processed_chunks[0] is, > > > > _concurrent/processed_chunks-824d746f-2a4d-48b6-8b51-e2534f1f86ca--array//elt-0 > > > Progress: time: Tue, 20 Nov 2012 15:38:04 -0800 Submitting:1 > > > Finished > > > successfully:1 > > > SwiftScript trace: Dim length = , 3.0 > > > SwiftScript trace: Splitting , 1.0 > > > > > > SwiftScript trace: Chunk number: , 1.0 > > > SwiftScript trace: Chunk number: , 2.0 > > > SwiftScript trace: Working furiously on , > > > > _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-1 > > > > > > SwiftScript trace: Splitting , 0.0 > > > SwiftScript trace: Chunk number: , 0.0 > > > SwiftScript trace: Splitting , 2.0 > > > SwiftScript trace: Working furiously on , > > > > _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-2 > > > SwiftScript trace: Working furiously on , > > > > _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-0 > > > Progress: time: Tue, 20 Nov 2012 15:38:05 -0800 Selecting site:1 > > > Stage > > > in:1 Finished successfully:3 > > > Progress: time: Tue, 20 Nov 2012 15:38:06 -0800 Checking status:1 > > > Finished successfully:4 > > > SwiftScript trace: Adding , 1.0 > > > SwiftScript trace: filename of output_chunks[i] is, > > > > _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-1 > > > SwiftScript trace: filename of output_chunks[i-1] is, > > > > _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-0 > > > SwiftScript trace: filename of processed_chunks[i] is, > > > > _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-1 > > > SwiftScript trace: Adding , 2.0 > > > SwiftScript trace: filename of processed_chunks[i] is, > > > > _concurrent/input_chunks-8bfa174c-e21b-4db2-9b58-ec96de60a1cc--array//elt-2 > > > SwiftScript trace: filename of output_chunks[i] is, > > > > _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-2 > > > SwiftScript trace: filename of output_chunks[i-1] is, > > > > _concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array//elt-1 > > > Execution failed: > > > File not found: > > > > /users/slic/tlai/work/mapReduceScripts/./_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array/_concurrent/output_chunks-33fce3ec-f16c-4430-b086-cc90f611a0a2--array/elt-2 > > > > > > > > > From the trace we can see filename of output_chunks[2] is > > > _concurrent/output_chunks- 33fce3ec-f16c-4430-b086- > > > cc90f611a0a2--array//elt-2 (we also verified the file is there after > > > the execution failure). > > > > > > > > > Thank you. > > > > > > On Tue, Nov 20, 2012 at 6:29 PM, Michael Wilde < wilde at mcs.anl.gov > > > > wrote: > > > > > > > > > Hao, > > > > > > Are you sure you assigned a value to output_chunks[2]? > > > > > > Can you post the entire program, and the entire error message? > > > > > > Thanks, > > > > > > - Mike > > > > > > > > > > > > ----- Original Message ----- > > > > From: "Hao Yang" < yanghao0614 at gmail.com > > > > > To: swift-user at ci.uchicago.edu > > > > Cc: mosastore at googlegroups.com > > > > Sent: Tuesday, November 20, 2012 7:22:45 PM > > > > Subject: [Swift-user] assign temporary file to mapper > > > > Hi, all: > > > > > > > > > > > > I have some problem assigning temporary file to some mapper > > > > variable > > > > (explained in the code below). > > > > > > > > > > > > type RSFile; > > > > string output_name; > > > > output_name = @arg("out","out.rsf"); > > > > RSFile output_data ; > > > > RSFile output_chunks[]; > > > > // some calculation to compute output_chunks[0], output_chunks[1], > > > > output_chunks[2] > > > > // at the end > > > > output_data = output_chunks[2]; > > > > > > > > > > > > > > > > When I try this script, swift reports " File output_chunks[2] not > > > > found " error and failed. But the output_chunks[2] is still > > > > available > > > > in the directory generated by swift ( _ concurrent/output_chunks- > > > > ae9bd0b3-ad88-45b0-95f4- f036d2d70e58--array). Should this error > > > > > > > message be expected? > > > > > > > > > > > > Thank you. > > > > > > > > -- > > > > Best Regards, > > > > Hao Yang > > > > > > > > The University of British Columbia > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > Swift-user mailing list > > > > Swift-user at ci.uchicago.edu > > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > > > > > -- > > > Michael Wilde > > > Computation Institute, University of Chicago > > > Mathematics and Computer Science Division > > > Argonne National Laboratory > > > > > > > > > > > > > > > > > > -- > > > Best Regards, > > > Hao Yang > > > Networked Systems Laboratory > > > Electrical and Computer Engineering > > > The University of British Columbia > > > > -- > > Michael Wilde > > Computation Institute, University of Chicago > > Mathematics and Computer Science Division > > Argonne National Laboratory > > > > > > > > > > > > -- > > Best Regards, > > Hao Yang > > Networked Systems Laboratory > > Electrical and Computer Engineering > > The University of British Columbia > > -- > Michael Wilde > Computation Institute, University of Chicago > Mathematics and Computer Science Division > Argonne National Laboratory > > -- Best Regards, Hao Yang Networked Systems Laboratory Electrical and Computer Engineering The University of British Columbia -------------- next part -------------- An HTML attachment was scrubbed... URL: