From jburge16 at my.whitworth.edu Mon Aug 10 15:06:18 2015 From: jburge16 at my.whitworth.edu (Jonathan S Burge) Date: Mon, 10 Aug 2015 20:06:18 +0000 Subject: [Swift-user] Installing swift on aws Message-ID: Currently I am trying to install swift-t on an instance of Amazon's Web Services. I am trying to install swift manually. The c-utils and adlb packages have compiled successfully, however, I am now having issues with turbine. Running ./configure and make seem to be functioning, but when make install is attempted, no directory is created and the user is prompted for another command without and verbose output. This is my command line for ./configure ./configure --prefix=/home/ubuntu/sfw/swift-t/turbine --with-adlb=$HOME/sfw/swift-t/lb --with-hdf5=no --enable-python=yes Running make check_includes shows that several things are not present, including mpi and python. However, I have already installed these and they make be either called or accessed through various commands. ex) python (opens python) mpichversion (returns version 3.0.4) If someone has advice or solutions to this problem, that would be greatly appreciated. Thanks. Jonathan -------------- next part -------------- An HTML attachment was scrubbed... URL: From jburge16 at my.whitworth.edu Mon Aug 10 15:18:13 2015 From: jburge16 at my.whitworth.edu (Jonathan S Burge) Date: Mon, 10 Aug 2015 20:18:13 +0000 Subject: [Swift-user] installing swift on aws Message-ID: Currently I am trying to install swift-t on an instance of Amazon's Web Services. I am trying to install swift manually. The c-utils and adlb packages have compiled successfully, however, I am now having issues with turbine. Running ./configure and make seem to be functioning, but when make install is attempted, no directory is created and the user is prompted for another command without and verbose output. This is my command line for ./configure ./configure --prefix=/home/ubuntu/sfw/swift-t/turbine --with-adlb=$HOME/sfw/swift-t/lb --with-hdf5=no --enable-python=yes Running make check_includes shows that several things are not present, including mpi and python. However, I have already installed these and they make be either called or accessed through various commands. ex) python (opens python) mpichversion (returns version 3.0.4) If someone has advice or solutions to this problem, that would be greatly appreciated. Thanks. Jonathan -------------- next part -------------- An HTML attachment was scrubbed... URL: From wozniak at mcs.anl.gov Mon Aug 10 15:43:14 2015 From: wozniak at mcs.anl.gov (Justin M Wozniak) Date: Mon, 10 Aug 2015 15:43:14 -0500 Subject: [Swift-user] installing swift on aws In-Reply-To: References: Message-ID: <55C90CE2.3050801@mcs.anl.gov> Hi Is this on Ubuntu? Justin On 08/10/2015 03:18 PM, Jonathan S Burge wrote: > > Currently I am trying to install swift-t on an instance of Amazon's > Web Services. I am trying to install swift manually. The c-utils and > adlb packages have compiled successfully, however, I am now having > issues with turbine. Running ./configure and make seem to be > functioning, but when make install is attempted, no directory is > created and the user is prompted for another command without and > verbose output. > > > This is my command line for ./configure > > ./configure --prefix=/home/ubuntu/sfw/swift-t/turbine > --with-adlb=$HOME/sfw/swift-t/lb --with-hdf5=no --enable-python=yes > > > Running make check_includes shows that several things are not present, > including mpi and python. > > However, I have already installed these and they make be either called > or accessed through various commands. > > ex) python (opens python) > > mpichversion (returns version 3.0.4) > > > If someone has advice or solutions to this problem, that would be > greatly appreciated. Thanks. > > > Jonathan > > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user -- Justin M Wozniak -------------- next part -------------- An HTML attachment was scrubbed... URL: From jenkins at mcs.anl.gov Tue Aug 11 13:53:34 2015 From: jenkins at mcs.anl.gov (Jenkins, Jonathan P.) Date: Tue, 11 Aug 2015 18:53:34 +0000 Subject: [Swift-user] 404 error in swift-t guide Message-ID: Good afternoon, I was perusing the swift-t user guide and tripped upon a 404, pointing to swift-lang.org/Swift-T/internals.html. Thanks, John From wozniak at mcs.anl.gov Wed Aug 12 14:12:43 2015 From: wozniak at mcs.anl.gov (Justin M Wozniak) Date: Wed, 12 Aug 2015 14:12:43 -0500 Subject: [Swift-user] 404 error in swift-t guide In-Reply-To: References: Message-ID: <55CB9AAB.7020903@mcs.anl.gov> Yes, the Turbine internals guide became greatly out-of-date- I need to remove that link. I can walk you through the internals in person if you like. On 08/11/2015 01:53 PM, Jenkins, Jonathan P. wrote: > Good afternoon, > > I was perusing the swift-t user guide and tripped upon a 404, pointing to > swift-lang.org/Swift-T/internals.html. > > Thanks, > John > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user -- Justin M Wozniak From jenkins at mcs.anl.gov Wed Aug 12 15:33:23 2015 From: jenkins at mcs.anl.gov (Jenkins, Jonathan P.) Date: Wed, 12 Aug 2015 20:33:23 +0000 Subject: [Swift-user] 404 error in swift-t guide In-Reply-To: <55CB9AAB.7020903@mcs.anl.gov> References: <55CB9AAB.7020903@mcs.anl.gov> Message-ID: Thanks for the offer! We should do that later on. On 8/12/15, 2:12 PM, "swift-user-bounces at ci.uchicago.edu on behalf of Justin M Wozniak" wrote: > >Yes, the Turbine internals guide became greatly out-of-date- I need to >remove that link. > >I can walk you through the internals in person if you like. > >On 08/11/2015 01:53 PM, Jenkins, Jonathan P. wrote: >> Good afternoon, >> >> I was perusing the swift-t user guide and tripped upon a 404, pointing >>to >> swift-lang.org/Swift-T/internals.html. >> >> Thanks, >> John >> >> _______________________________________________ >> Swift-user mailing list >> Swift-user at ci.uchicago.edu >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > >-- >Justin M Wozniak > >_______________________________________________ >Swift-user mailing list >Swift-user at ci.uchicago.edu >https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user From pw410 at cam.ac.uk Fri Aug 21 08:19:20 2015 From: pw410 at cam.ac.uk (Peter Wortmann) Date: Fri, 21 Aug 2015 14:19:20 +0100 Subject: [Swift-user] Too many open file descriptors Message-ID: <55D72558.8050504@cam.ac.uk> Hi swift-user We are using Swift/T in a way that does a lot of "@stdout=..." indirections. However, this seems to lead to Turbine running out of file descriptors at a certain point. A bit of poking showed that this calls through to "mktemp" in "files.tcl", which states: # This can fail with "too many open files" file tempfile result $tmpdir/.turbine Is this a known bug? I tried to work around it, and simply closing the returned file channel seems to work: close [file tempfile result $tmpdir/.turbine] Is there a reason to keep it open? Greetings, Peter From wozniak at mcs.anl.gov Fri Aug 21 14:42:01 2015 From: wozniak at mcs.anl.gov (Justin M Wozniak) Date: Fri, 21 Aug 2015 14:42:01 -0500 Subject: [Swift-user] Too many open file descriptors In-Reply-To: <55D72558.8050504@cam.ac.uk> References: <55D72558.8050504@cam.ac.uk> Message-ID: <55D77F09.2060805@mcs.anl.gov> Hi Sorry about that- it looks like we missed the point in the docs for this new Tcl feature that it returns an open file descriptor. We will close it as you suggest. Thanks Justin On 08/21/2015 08:19 AM, Peter Wortmann wrote: > Hi swift-user > > We are using Swift/T in a way that does a lot of "@stdout=..." > indirections. However, this seems to lead to Turbine running out of file > descriptors at a certain point. A bit of poking showed that this calls > through to "mktemp" in "files.tcl", which states: > > # This can fail with "too many open files" > file tempfile result $tmpdir/.turbine > > Is this a known bug? I tried to work around it, and simply closing the > returned file channel seems to work: > > close [file tempfile result $tmpdir/.turbine] > > Is there a reason to keep it open? > > Greetings, > Peter > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user -- Justin M Wozniak From tim.g.armstrong at gmail.com Fri Aug 21 15:44:41 2015 From: tim.g.armstrong at gmail.com (Tim Armstrong) Date: Fri, 21 Aug 2015 13:44:41 -0700 Subject: [Swift-user] Too many open file descriptors In-Reply-To: <55D77F09.2060805@mcs.anl.gov> References: <55D72558.8050504@cam.ac.uk> <55D77F09.2060805@mcs.anl.gov> Message-ID: Nice catch! On 21 August 2015 at 12:42, Justin M Wozniak wrote: > Hi > Sorry about that- it looks like we missed the point in the docs for > this new Tcl feature that it returns an open file descriptor. We will > close it as you suggest. > Thanks > Justin > > On 08/21/2015 08:19 AM, Peter Wortmann wrote: > > Hi swift-user > > > > We are using Swift/T in a way that does a lot of "@stdout=..." > > indirections. However, this seems to lead to Turbine running out of file > > descriptors at a certain point. A bit of poking showed that this calls > > through to "mktemp" in "files.tcl", which states: > > > > # This can fail with "too many open files" > > file tempfile result $tmpdir/.turbine > > > > Is this a known bug? I tried to work around it, and simply closing the > > returned file channel seems to work: > > > > close [file tempfile result $tmpdir/.turbine] > > > > Is there a reason to keep it open? > > > > Greetings, > > Peter > > > > _______________________________________________ > > Swift-user mailing list > > Swift-user at ci.uchicago.edu > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > > -- > Justin M Wozniak > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pw410 at cam.ac.uk Mon Aug 24 06:54:10 2015 From: pw410 at cam.ac.uk (Peter Wortmann) Date: Mon, 24 Aug 2015 12:54:10 +0100 Subject: [Swift-user] Too many open file descriptors In-Reply-To: <55D77F09.2060805@mcs.anl.gov> References: <55D72558.8050504@cam.ac.uk> <55D77F09.2060805@mcs.anl.gov> Message-ID: <55DB05E2.7000200@cam.ac.uk> Hi Okay, I was a bit unsure - the comment made it look deliberate. Thanks for the confirmation! Greetings, Peter On 21/08/15 20:42, Justin M Wozniak wrote: > Hi > Sorry about that- it looks like we missed the point in the docs for > this new Tcl feature that it returns an open file descriptor. We will > close it as you suggest. > Thanks > Justin > > On 08/21/2015 08:19 AM, Peter Wortmann wrote: >> Hi swift-user >> >> We are using Swift/T in a way that does a lot of "@stdout=..." >> indirections. However, this seems to lead to Turbine running out of file >> descriptors at a certain point. A bit of poking showed that this calls >> through to "mktemp" in "files.tcl", which states: >> >> # This can fail with "too many open files" >> file tempfile result $tmpdir/.turbine >> >> Is this a known bug? I tried to work around it, and simply closing the >> returned file channel seems to work: >> >> close [file tempfile result $tmpdir/.turbine] >> >> Is there a reason to keep it open? >> >> Greetings, >> Peter >> >> _______________________________________________ >> Swift-user mailing list >> Swift-user at ci.uchicago.edu >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > From iraicu at cs.iit.edu Mon Aug 31 09:54:16 2015 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Mon, 31 Aug 2015 10:54:16 -0400 Subject: [Swift-user] CFP: 8th Workshop on Many-Task Computing on Clouds, Grids, and Supercomputers (MTAGS) 2015 -- co-located with IEEE/ACM Supercomputing/SC 2015 Message-ID: <55E46A98.8070504@cs.iit.edu> Call for Papers --------------------------------------------------------------------------------------- The 8th Workshop on Many-Task Computing on Clouds, Grids, and Supercomputers (MTAGS) 2015 http://datasys.cs.iit.edu/events/MTAGS15/ --------------------------------------------------------------------------------------- November 15th, 2015 Austin, Texas, USA Co-located with with IEEE/ACM International Conference for High Performance Computing, Networking, Storage and Analysis (SC15) ======================================================================================= The 8th workshop on Many-Task Computing on Clouds, Grids, and Supercomputers (MTAGS) will provide the scientific community a dedicated forum for presenting new research, development, and deployment efforts of large-scale many-task computing (MTC) applications on large scale clusters, Grids, Supercomputers, and Cloud Computing infrastructure. MTC, the theme of the workshop encompasses loosely coupled applications, which are generally composed of many tasks (both independent and dependent tasks) to achieve some larger application goal. This workshop will cover challenges that can hamper efficiency and utilization in running applications on large-scale systems, such as local resource manager scalability and granularity, efficient utilization of raw hardware, parallel file system contention and scalability, data management, I/O management, reliability at scale, and application scalability. We welcome paper submissions on all theoretical, simulations, and systems topics related to MTC, but we give special consideration to papers addressing petascale to exascale challenges. Papers will be peer-reviewed for novelty, scientific merit, and scope for the workshop. The workshop will be co-located with the IEEE/ACM Supercomputing 2015 Conference in Austin Texas on November 15th, 2015. For more information, please see http://datasys.cs.iit.edu/events/MTAGS15/. For more information on past workshops, please see MTAGS14, MTAGS13, MTAGS12, MTAGS11, MTAGS10, MTAGS09, and MTAGS08. We also ran a Special Issue on Many-Task Computing in the IEEE Transactions on Parallel and Distributed Systems (TPDS) which appeared in June 2011; the proceedings can be found online at http://www.computer.org/portal/web/csdl/abs/trans/td/2011/06/ttd201106toc.htm. We are also currently assembling a new special issue in the IEEE Transaction on Cloud Computing on Many-Task Computing in the Cloud, see http://datasys.cs.iit.edu/events/TCC-MTC15/index.html for more information. We, the workshop organizers, also published a highly relevant paper that defines Many-Task Computing which was published in MTAGS08,titled ?Many-Task Computing for Grids and Supercomputers?; we encourage potential authors to read this paper, and to clearly articulate in your paper submissions how your papers are related to Many-Task Computing. Topics --------------------------------------------------------------------------------------- We invite the submission of original work that is related to the topics below. The papers should be 6 pages, including all figures and references.We aim to cover topics related to Many-Task Computing on each of the three major distributed systems paradigms, Cloud Computing, Grid Computing and Supercomputing. Topics of interest include: * Compute Resource Management * Scheduling * Job execution frameworks * Local resource manager extensions * Performance evaluation of resource managers in use on large scale systems * Dynamic resource provisioning * Techniques to manage many-core resources and/or GPUs * Challenges and opportunities in running many-task workloads on HPC systems * Challenges and opportunities in running many-task workloads on Cloud Computing infrastructure * Storage architectures and implementations * Distributed file systems * Parallel file systems * Distributed meta-data management * Content distribution systems for large data * Data caching frameworks and techniques * Data management within and across data centers * Data-aware scheduling * Data-intensive computing applications * Eventual-consistency storage usage and management * Programming models and tools * Map-reduce and its generalizations * Many-task computing middleware and applications * Parallel programming frameworks * Ensemble MPI techniques and frameworks * Service-oriented science applications * Large-Scale Workflow Systems * Workflow system performance and scalability analysis * Scalability of workflow systems * Workflow infrastructure and e-Science middleware * Programming Paradigms and Models * Large-Scale Many-Task Applications * High-throughput computing (HTC) applications * Data-intensive applications * Quasi-supercomputing applications, deployments, and experiences * Performance Evaluation * Performance evaluation * Real systems * Simulations * Reliability of large systems Paper Submission and Publication --------------------------------------------------------------------------------------- Authors are invited to submit papers with unpublished, original work of not more than 6 pages of double column text using single spaced 10 point size on 8.5 x 11 inch pages, as per ACM 8.5 x 11 manuscript guidelines; document templates can be found at http://www.acm.org/sigs/publications/proceedings-templates. The final 6 page papers in PDF format must be submitted online at https://cmt.research.microsoft.com/MTAGS2015/ before the deadline. Papers will be peer-reviewed for novelty, scientific merit, and scope for the workshop. Submission implies the willingness of at least one of the authors to register and present the paper. For more information, please see http://datasys.cs.iit.edu/events/MTAGS15/. Important Dates --------------------------------------------------------------------------------------- * Abstracts due: September 4th, 2015 * Full paper due: September 11th, 2015 * Acceptance notification: October 2nd, 2015 * Early Registration deadline: October 15th, 2015 * Final papers due: November 6th, 2015 * Workshop date: November 15th, 2015 Committee Members --------------------------------------------------------------------------------------- Workshop Chairs * Ioan Raicu, Illinois Institute of Technology & Argonne National Laboratory * Justin Wozniak, University of Chicago & Argonne National Laboratory * Ian Foster, University of Chicago & Argonne National Laboratory * Yong Zhao, University of Electronic Science and Technology of China Steering Committee * David Abramson, Monash University, Australia * Jack Dongarra, University of Tennessee, USA * Geoffrey Fox, Indiana University, USA * Manish Parashar, Rutgers University, USA * Marc Snir, University of Illinois at Urbana Champaign, USA * Xian-He Sun, Illinois Institute of Technology, USA * Weimin Zheng, Tsinghua University, China Technical Committee * Hasan Abbasi, Oak Ridge National Labs, USA * Tim Armstrong, University of Chicago, USA * Roger Barga, Microsoft, USA * Mihai Budiu, Microsoft Research, USA * Rajkumar Buyya University of Melbourne, Australia * Kyle Chard, University of Chicago, USA * Evangelinos Constantinos, Massachusetts Institute of Technology, USA * Catalin Dumitrescu, Fermi National Labs, USA * Haryadi Gunawi, University of Chicago, USA * Indranil Gupta, University of Illinois at Urbana Champaign, USA * Alexandru Iosup, Delft University of Technology, Netherlands * Florin Isaila, Universidad Carlos III de Madrid, Spain & Argonne National Laboratory, USA * Kamil Iskra, Argonne National Laboratory, USA * Daniel S. Katz, University of Chicago, USA * Jik-Soo Kim, Kristi, Korea * Scott A. Klasky, Oak Ridge National Labs, USA * Mike Lang, Los Alamos National Laboratory, USA * Tonglin Li, Illinois Institute of Technology, USA * Chris Moretti, Princeton University, USA * David O'Hallaron, Carnegie Mellon University, USA * Marlon Pierce, Indiana University, USA * Judy Qiu, Indiana University, USA * Lavanya Ramakrishnan, Lawrence Berkeley National Laboratory, USA * Matei Ripeanu, University of British Columbia, Canada * Iman Sadooghi, Illinois Institute of Technology, USA * Wei Tang, Argonne National Laboratory, USA * Edward Walker, Whitworth University, USA * Ke Wang, Illinois Institute of Technology, USA * Matthew Woitaszek, Walmart Labs, USA * Rich Wolski, University of California, Santa Barbara, USA * Zhifeng Yun, University of Houston, USA * Zhao Zhang, University of California at Berkeley, USA * Ziming Zheng, University of Chicago, USA -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Editor: IEEE TCC, Springer Cluster, Springer JoCCASA Chair: IEEE/ACM MTAGS, ACM ScienceCloud ================================================================= Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ LinkedIn: http://www.linkedin.com/in/ioanraicu Google: http://scholar.google.com/citations?user=jE73HYAAAAAJ ================================================================= =================================================================