From iraicu at cs.iit.edu Sat Jan 5 09:11:58 2013 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Sat, 05 Jan 2013 09:11:58 -0600 Subject: [Swift-devel] CFP: Scientific Cloud Computing (ScienceCloud) -- co-located with ACM HPDC 2013 Message-ID: <50E842BE.20307@cs.iit.edu> ------------------------------------------------------------------------------- *** Call for Papers *** 4th Workshop on Scientific Cloud Computing (ScienceCloud) 2013 Co-located with ACM HPDC 2013, New York City, NY, USA -- June 17th, 2013 http://datasys.cs.iit.edu/events/ScienceCloud2013/ ------------------------------------------------------------------------------- Computational and Data-Driven Sciences have become the third and fourth pillar of scientific discovery in addition to experimental and theoretical sciences. Scientific Computing has already begun to change how science is done, enabling scientific breakthroughs through new kinds of experiments that would have been impossible only a decade ago. Today?s ?Big Data? science is generating datasets that are increasing exponentially in both complexity and volume, making their analysis, archival, and sharing one of the grand challenges of the 21st century. The support for data intensive computing is critical to advance modern science as storage systems have exposed a widening gap between their capacity and their bandwidth by more than 10-fold over the last decade. There is a growing need for advanced techniques to manipulate, visualize and interpret large datasets. Scientific Computing is the key to solving ?grand challenges? in many domains and providing breakthroughs in new knowledge, and it comes in many shapes and forms: high-performance computing (HPC) which is heavily focused on compute- intensive applications; high-throughput computing (HTC) which focuses on using many computing resources over long periods of time to accomplish its computational tasks; many-task computing (MTC) which aims to bridge the gap between HPC and HTC by focusing on using many resources over short periods of time; and data-intensive computing which is heavily focused on data distribution, data-parallel execution, and harnessing data locality by scheduling of computations close to the data. The 4th workshop on Scientific Cloud Computing (ScienceCloud) will provide the scientific community a dedicated forum for discussing new research, development, and deployment efforts in running these kinds of scientific computing workloads on Cloud Computing infrastructures. The ScienceCloud workshop will focus on the use of cloud-based technologies to meet new compute-intensive and data- intensive scientific challenges that are not well served by the current supercomputers, grids and HPC clusters. The workshop will aim to address questions such as: What architectural changes to the current cloud frameworks (hardware, operating systems, networking and/or programming models) are needed to support science? Dynamic information derived from remote instruments and coupled simulation, and sensor ensembles that stream data for real-time analysis are important emerging techniques in scientific and cyber-physical engineering systems. How can cloud technologies enable and adapt to these new scientific approaches dealing with dynamism? How are scientists using clouds? Are there scientific HPC/HTC/MTC workloads that are suitable candidates to take advantage of emerging cloud computing resources with high efficiency? Commercial public clouds provide easy access to cloud infrastructure for scientists. What are the gaps in commercial cloud offerings and how can they be adapted for running existing and novel eScience applications? What benefits exist by adopting the cloud model, over clusters, grids, or supercomputers? What factors are limiting clouds use or would make them more usable/efficient? This workshop encourages interaction and cross-pollination between those developing applications, algorithms, software, hardware and networking, emphasizing scientific computing for such cloud platforms. We believe the workshop will be an excellent place to help the community define the current state, determine future goals, and define architectures and services for future science clouds. TOPICS ------------------------------------------------------------------------------- We invite the submission of original work that is related to the topics below. The papers can be either short (4 pages) position papers, or long (8 pages) research papers. Topics of interest include (in the context of Cloud Computing): - Scientific application cases studies on Cloud infrastructure - Performance evaluation of Cloud environments and technologies - Fault tolerance and reliability in cloud systems - Data-intensive workloads and tools on Clouds - Use of programming models such as Map-Reduce and its implementations - Storage cloud architectures - I/O and Data management in the Cloud - Workflow and resource management in the Cloud - Use of cloud technologies (e.g., NoSQL databases) for scientific applications - Data streaming and dynamic applications on Clouds - Dynamic resource provisioning - Many-Task Computing in the Cloud - Application of cloud concepts in HPC environments or vice versa - High performance parallel file systems in virtual environments - Virtualized high performance I/O network interconnects - Virtualization - Distributed Operating Systems - Many-core computing and accelerators (e.g. GPUs, MIC) in the Cloud - Cloud security IMPORTANT DATES ------------------------------------------------------------------------------- - Paper submission: February 11th, 2013 (11:59PM PST) - Acceptance notification: March 18th, 2013 - Final papers due: April 15th, 2013 PAPER SUBMISSION ------------------------------------------------------------------------------- Authors are invited to submit papers with unpublished, original work of not more than 8 pages of double column text using single spaced 10 point size on 8.5 x 11 inch pages (including all text, figures, and references), as per ACM 8.5 x 11 manuscript guidelines (document templates can be found at http://www.acm.org/sigs/publications/proceedings-templates). A 250 word abstract and the final paper in PDF format must be submitted online at https://cmt.research.microsoft.com/ScienceCloud2013/ before the deadline of February 11th, 2013 at 11:59PM PST. Papers will be peer-reviewed, and accepted papers will be published in the workshop proceedings as part of the ACM digital library. Notifications of the paper decisions will be sent out by March 18th, 2013. Selected excellent work will be invited to submit extended versions of the workshop paper to a special issue journal. Submission implies the willingness of at least one of the authors to register and present the paper. GENERAL CHAIRS ------------------------------------------------------------------------------- - Ioan Raicu, Illinois Institute of Technology & Argonne National Lab., USA - Yogesh Simmhan, University of Southern California, USA PROGRAM COMMITTEE CHAIRS ------------------------------------------------------------------------------- - Kyle Chard, University of Chicago, USA - Gabriel Antoniu, INRIA, France - Lavanya Ramakrishnan, Lawrence Berkeley National Lab, USA STEERING COMMITTEE ------------------------------------------------------------------------------- - Ian Foster, University of Chicago & Argonne National Laboratory, USA - Pete Beckman, University of Chicago & Argonne National Laboratory, USA - Carole Goble, University of Manchester, UK - Dennis Gannon, Microsoft Research, USA - Robert Grossman, University of Chicago, USA - Kate Keahey, University of Chicago & Argonne National Laboratory, USA - Ed Lazowska, University of Washington & Computing Community Consortium, USA - David O'Hallaron, Carnegie Mellon University & Intel Labs, USA - Jack Dongarra, University of Tennessee, USA - Geoffrey Fox, Indiana University, USA PROGRAM COMMITTEE ------------------------------------------------------------------------------- - Samer Al-Kiswany (University of British Columbia) - Roger Barga (Microsoft Research) - Roy Campbell (University of Illinois at Urbana Champaign) - Charlie Catlett (Argonne National Laboratory) - Simon Caton (KIT) - David Chiu (Washington State University) - Jack Dongara (University of Tennessee) - Ake Edlund (Royal Institute of Technology) - Chathura Herath (Indiana University) - Neil Chue Hong (University of Edinburgh) - Adriana Iamnitchi (University of South Florida) - Shantenu Jha (Louisiana State University) - Hui Jin (Illinois Institute of Technology) - Carl Kesselman (University of Southern California) - Thilo Kielmann (Vrije University) - Gregor von Laszewski (Indiana University) - Shiyong Lu (Wayne State University) - Wei Lu (Microsoft Research) - Andr Luckow (Louisiana State University) - David Martin (Argonne National Laboratory) - Gabriel Mateescu (Virginia Tech) - Paolo Missier (University of Manchester) - Ruben Montero (Universidad Complutense de Madrid) - Reagan Moore (University of North Carolina) - Jose Moreira (IBM Research) - Christine Morin (INRIA Rennes) - Pasquale Pagano (ISTI) - Beth Plale (Indiana University) - Omer Rana (Cardiff University) - Matei Ripeanu (University of British Columbia) - Josh Simons (VMWare) - Douglas Thain (University of Notre Dame) - Johan Tordsson (Ume University) - Vasudeva Varma (IIIT-Hyderabad) - Zhifeng Yun (Louisiana State University) - Yong Zhao (University of Electronic and Science Technology of China) -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ ================================================================= ================================================================= From marialemos72 at gmail.com Sat Jan 5 12:59:59 2013 From: marialemos72 at gmail.com (Maria Lemos) Date: Sat, 5 Jan 2013 18:59:59 +0000 Subject: [Swift-devel] Accepted Workshops in the CISTI'2013 - 8th Iberian Conference on IST Message-ID: <20130105185858.83C567CC095@mailrelay.anl.gov> *************************************************************************************************** CISTI'2013 Accepted Workshops 8th Iberian Conference on Information Systems and Technologies Lisbon, Portugal, June 19 - 23, 2013 http://www.aisti.eu/cisti2013/index.php?option=com_content&view=article&id=64&Itemid=68&lang=en *************************************************************************************************** List of accepted workshops in the CISTI'2013 - 8th Iberian Conference on Information Systems and Technologies: > IAwDQ'2013 - Fourth Ibero-American Workshop on Data Quality > SGaMePlay'2013 - Third Iberian Workshop on Serious Games and Meaningful Play > WISA'2013 - Fifth Workshop on Intelligent Systems and Applications > WISIS'2013 - Third Workshop on Information Systems for Interactive Spaces > WSEQP'2013 - First Workshop in Software Engineering and Quality Process Best regards, Maria Lemos AISTI / CISTI'2013 http://www.aisti.eu/cisti2013 From iraicu at cs.iit.edu Sun Jan 6 07:54:14 2013 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Sun, 06 Jan 2013 07:54:14 -0600 Subject: [Swift-devel] CFP: ACM HPDC 2013 -- Abstracts due January 14th Message-ID: <50E98206.5040308@cs.iit.edu> **** CALL FOR PAPERS **** The 22nd International ACM Symposium on High-Performance Parallel and Distributed Computing (HPDC'13) New York City, USA - June 17-21, 2013 http://www.hpdc.org/2013 The ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) is the premier annual conference for presenting the latest research on the design, implementation, evaluation, and the use of parallel and distributed systems for high-end computing. In 2013, the 22nd HPDC and affiliated workshops will take place in the heart of iconic New York City from June 17-21. **** IMPORTANT DATES **** Abstracts: 14 January 2013 Papers: 21 January 2013 (no extensions) Reviews released to authors: 6 March 2013 Author rebuttals due: 10 March 2013 Author notifications: 17 March 2013 **** HPDC'13 GENERAL CO-CHAIRS **** Manish Parashar, Rutgers University Jon Weissman, University of Minnesota **** HPDC'13 PROGRAM CO-CHAIRS **** Dick Epema, Delft University of Technology Renato Figueiredo, University of Florida **** HPDC'13 WORKSHOPS CHAIR **** Abhishek Chandra, University of Minnesota **** HPDC'13 LOCAL ARRANGEMENTS CHAIR **** Daniele Scarpazza, DEShaw Research **** HPDC'13 SPONSORSHIP CHAIR **** Dean Hildebrand, IBM Almaden **** HPDC'13 PUBLICITY CO-CHAIRS **** Alexandru Iosup, Delft University of Technology, the Netherlands Ioan Raicu, Illinois Institute of Technology, USA Kenjiro Taura, University of Tokyo, Japan Bruno Schulze, National Laboratory for Scientific Computing, Brazil **** SCOPE AND TOPICS **** Submissions are welcomed on high-performance parallel and distributed computing topics including but not limited to: clusters, clouds, grids, data-intensive computing, massively multicore, and global-scale computing systems. New scholarly research showing empirical and reproducible results in architectures, systems, and networks is strongly encouraged, as are experience reports of operational deployments that can provide insights for future research on HPDC applications and systems. All papers will be evaluated for their originality, technical depth and correctness, potential impact, relevance to the conference, and quality of presentation. Research papers must clearly demonstrate research contributions and novelty, while experience reports must clearly describe lessons learned and demonstrate impact. In the context of high-performance parallel and distributed computing, the topics of interest include, but are not limited to: * Systems, networks, and architectures for high-end computing * Massively multicore systems * Resource virtualization * Programming languages and environments * I/O, storage systems, and data management * Resource management and scheduling, including energy-?????aware techniques * Performance modeling and analysis * Fault tolerance, reliability, and availability * Data-intensive computing * Applications of parallel and distributed computing **** PAPER SUBMISSION GUIDELINES **** Authors are invited to submit technical papers of at most 12 pages in PDF format, including figures and references. Papers should be formatted in the ACM Proceedings Style and submitted via the conference web site. No changes to the margins, spacing, or font sizes as specified by the style file are allowed. Accepted papers will appear in the conference proceedings, and will be incorporated into the ACM Digital Library. A limited number of papers will be accepted as posters. Papers must be self-contained and provide the technical substance required for the program committee to evaluate their contributions. Papers should thoughtfully address all related work, particularly work presented at previous HPDC events. Submitted papers must be original work that has not appeared in and is not under consideration for another conference or a journal. See the ACM Prior Publication Policy for more details. **** IMPORTANT DATES **** Abstracts Due: 14 January 2013 Papers Due: 21 January 2013 (no extensions) **** Program Committee **** David Abramson, Monash University, Australia Kento Aida, National Institute of Informatics, Japan Gabriel Antoniu INRIA, France Henri Bal, Vrije Universiteit, the Netherlands Adam Barker, University of St Andrews, UK Michela Becchi, University of Missouri - Columbia, USA John Bent, EMC, USA Ali Butt, Virginia Tech, USA Kirk Cameron, Virginia Tech, USA Franck Cappello, INRIA, France and University of Illinois at Urbana-Champaign, USA Henri Casanova, University of Hawaii, USA Abhishek Chandra, University of Minnesota, USA Andrew Chien, University of Chicago and Argonne National Laboratory, USA Paolo Costa, Imperial College London, UK Peter Dinda, Northwestern University, USA Gilles Fedak, INRIA, France Ian Foster, University of Chicago and Argonne National Laboratory, USA Clemens Grelck, University of Amsterdam, the Netherlands Dean Hildebrand, IBM Research, USA Fabrice Huet, INRIA-University of Nice, France Adriana Iamnitchi, University of South Florida, USA Alexandru Iosup, Delft University of Technology, the Netherlands Kate Keahey, Argonne National Laboratory, USA Thilo Kielmann, Vrije Universiteit, the Netherlands Charles Kilian, Purdue University, USA Zhiling Lan, Illinois Institute of Technology, USA John Lange, University of Pittsburgh, USA Barney Maccabe, Oak Ridge National Laboratory, USA Carlos Maltzahn, University of California, Santa Cruz, USA Naoya Maruyama, RIKEN Advanced Institute for Computational Science, Japan Satoshi Matsuoka, Tokyo Institute of Technology, Japan Manish Parashar, Rutgers University, USA Judy Qiu, Indiana University, USA Ioan Raicu, Illinois Institute of Technology, USA Philip Rhodes, University of Mississippi, USA Matei Ripeanu, University of British Columbia, Canada Prasenjit Sarkar, IBM Research, USA Daniele Scarpazza, D.E. Shaw Research, USA Karsten Schwan, Georgia Institute of Technology, USA Martin Swany, Indiana University, USA Michela Taufer, University of Delaware, USA Kenjiro Taura, University of Tokyo, Japan Douglas Thain, University of Notre Dame, USA Cristian Ungureanu, NEC Research, USA Ana Varbanescu, Delft University of Technology, the Netherlands Chuliang Weng, Shanghai Jiao Tong University, China Jon Weissman, University of Minnesota, USA Yongwei Wu, Tsinghua University, China Dongyan Xu, Purdue University, USA Ming Zhao, Florida International University, USA **** Steering Committee **** Henri Bal, Vrije Universiteit, the Netherlands Andrew A. Chien, University of Chicago and Argonne National Laboratory, USA Peter Dinda, Northwestern University, USA Dick Epema, Delft University of Technology, the Netherlands Ian Foster, University of Chicago and Argonne National Laboratory, USA Salim Hariri, University of Arizona, USA Thilo Kielmann, Vrije Universiteit, the Netherlands Arthur "Barney" Maccabe, Oak Ridge National Laboratory, USA Manish Parashar, Rutgers University, USA Matei Ripeanu, University of British Columbia, Canada Karsten Schwan, Georgia Tech, USA Doug Thain, University of Notre Dame, USA Jon Weissman, University of Minnesota (Chair), USA -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ ================================================================= ================================================================= From davidk at ci.uchicago.edu Sun Jan 6 16:55:20 2013 From: davidk at ci.uchicago.edu (David Kelly) Date: Sun, 6 Jan 2013 16:55:20 -0600 (CST) Subject: [Swift-devel] swift.log In-Reply-To: <169903500.15054.1357512908839.JavaMail.root@zimbra-mb2.anl.gov> Message-ID: <1745806218.15056.1357512920366.JavaMail.root@zimbra-mb2.anl.gov> Is swift.log still useful at this point? It usually only contains one or two lines along the lines of: 2013-01-06 22:49:11,118+0000 DEBUG Loader Swift started It seems like everything useful ends up in appname.log. Is there any advantage to keeping swift.log around? David From hategan at mcs.anl.gov Sun Jan 6 17:05:23 2013 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sun, 06 Jan 2013 15:05:23 -0800 Subject: [Swift-devel] swift.log In-Reply-To: <1745806218.15056.1357512920366.JavaMail.root@zimbra-mb2.anl.gov> References: <1745806218.15056.1357512920366.JavaMail.root@zimbra-mb2.anl.gov> Message-ID: <1357513523.21942.2.camel@blabla> I don't think it was ever useful. It's there because it's the default log4j log file name. If there are any logging statements before swift gets to configure the proper log file name, they make it to swift.log. I think we should find a way to make sure that swift.log is not created because it might be confusing, but let's discuss it first. Mihael On Sun, 2013-01-06 at 16:55 -0600, David Kelly wrote: > Is swift.log still useful at this point? It usually only contains one or two lines along the lines of: > > 2013-01-06 22:49:11,118+0000 DEBUG Loader Swift started > > It seems like everything useful ends up in appname.log. Is there any advantage to keeping swift.log around? > > David > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel From lpesce at uchicago.edu Mon Jan 7 08:40:25 2013 From: lpesce at uchicago.edu (Lorenzo Pesce) Date: Mon, 7 Jan 2013 08:40:25 -0600 Subject: [Swift-devel] swift.log In-Reply-To: <1357513523.21942.2.camel@blabla> References: <1745806218.15056.1357512920366.JavaMail.root@zimbra-mb2.anl.gov> <1357513523.21942.2.camel@blabla> Message-ID: <2CD6DD2E-D7D0-4CA1-867D-FC141AB977C7@uchicago.edu> I see no use in it. Users usually make jokes about it. On Jan 6, 2013, at 5:05 PM, Mihael Hategan wrote: > I don't think it was ever useful. It's there because it's the default > log4j log file name. If there are any logging statements before swift > gets to configure the proper log file name, they make it to swift.log. > > I think we should find a way to make sure that swift.log is not created > because it might be confusing, but let's discuss it first. > > Mihael > > On Sun, 2013-01-06 at 16:55 -0600, David Kelly wrote: >> Is swift.log still useful at this point? It usually only contains one or two lines along the lines of: >> >> 2013-01-06 22:49:11,118+0000 DEBUG Loader Swift started >> >> It seems like everything useful ends up in appname.log. Is there any advantage to keeping swift.log around? >> >> David >> _______________________________________________ >> Swift-devel mailing list >> Swift-devel at ci.uchicago.edu >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel From davidk at ci.uchicago.edu Mon Jan 14 16:54:25 2013 From: davidk at ci.uchicago.edu (David Kelly) Date: Mon, 14 Jan 2013 16:54:25 -0600 (CST) Subject: [Swift-devel] Question about caching In-Reply-To: <1279905120.59832.1358202778986.JavaMail.root@zimbra-mb2.anl.gov> Message-ID: <1683516474.60030.1358204065205.JavaMail.root@zimbra-mb2.anl.gov> I am working with an application that uses coaster provider staging to stage files. One of the files, common.tar.gz, gets passed to every app call. Does this file actually get transferred once per task, or does each instance of worker.pl keep a cached copy of common.tar.gz on the node? From hategan at mcs.anl.gov Mon Jan 14 19:00:59 2013 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 14 Jan 2013 17:00:59 -0800 Subject: [Swift-devel] Question about caching In-Reply-To: <1683516474.60030.1358204065205.JavaMail.root@zimbra-mb2.anl.gov> References: <1683516474.60030.1358204065205.JavaMail.root@zimbra-mb2.anl.gov> Message-ID: <1358211659.1435.0.camel@blabla> There is no caching at the coaster level at this time, and if you use provider staging, swift doesn't do any caching either, so no. Mihael On Mon, 2013-01-14 at 16:54 -0600, David Kelly wrote: > I am working with an application that uses coaster provider staging to stage files. One of the files, common.tar.gz, gets passed to every app call. Does this file actually get transferred once per task, or does each instance of worker.pl keep a cached copy of common.tar.gz on the node? > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel From wilde at mcs.anl.gov Thu Jan 17 10:48:09 2013 From: wilde at mcs.anl.gov (Michael Wilde) Date: Thu, 17 Jan 2013 10:48:09 -0600 (CST) Subject: [Swift-devel] Interesting paper on measuring Condor usage Message-ID: <358627550.4155.1358441289950.JavaMail.root@zimbra.anl.gov> Possibly some lessons for measuring Swift usage: http://www.cse.nd.edu/~dthain/papers/measure-ccpe.pdf - Mike From marialemos72 at gmail.com Tue Jan 22 13:22:18 2013 From: marialemos72 at gmail.com (Maria Lemos) Date: Tue, 22 Jan 2013 19:22:18 +0000 Subject: [Swift-devel] Workshops in the CISTI'2013 - 8th Iberian Conference on IST Message-ID: <20130122192112.35F427CC0B1@mailrelay.anl.gov> *************************************************************************************************** Workshop in the CISTI'2013 8th Iberian Conference on Information Systems and Technologies Lisbon, Portugal, June 19 - 23, 2013 http://www.aisti.eu/cisti2013/index.php?option=com_content&view=article&id=82&Itemid=67&lang=en *************************************************************************************************** Complete list of workshops accepted in the CISTI'2013 - 8th Iberian Conference on Information Systems and Technologies: > IAwDQ 2013 - Fourth Ibero-American Workshop on Data Quality > SGaMePlay 2013 - Third Iberian Workshop on Serious Games and Meaningful Play > TICAMES 2013 - First Workshop on Information and Communication Technology in Higher Education: Learning Mathematics > WIA 2013 - Primero Workshop en Innovaci?n Abierta > WISA 2013 - Fifth Workshop on Intelligent Systems and Applications > WISIS 2013 - Third Workshop on Information Systems for Interactive Spaces > WSEQP 2013 - First Workshop in Software Engineering and Quality Process Best regards, Maria Lemos AISTI / CISTI'2013 http://www.aisti.eu/cisti2013 From davidk at ci.uchicago.edu Thu Jan 24 07:25:39 2013 From: davidk at ci.uchicago.edu (David Kelly) Date: Thu, 24 Jan 2013 07:25:39 -0600 (CST) Subject: [Swift-devel] Why is Swift throttling at 1024 tasks? In-Reply-To: <456675298.1346868.1359032680809.JavaMail.root@ci.uchicago.edu> Message-ID: <1348652026.1348480.1359033939697.JavaMail.root@ci.uchicago.edu> Hello all, I am trying to run DSSAT using 4096 cores, but I can't seem to get Swift to go beyond 1024 active tasks. Looking at the coaster log, I see all 4096 connecting. I am using persistent coasters, with both Swift and the coaster service running on the same node. I have my job throttle set to 4000. I also tried explicitly setting foreach.max.threads to 4096. I checked ulimit and have a limit of 4096 open files. Is there some other limit I am missing? http://www.ci.uchicago.edu/~davidk/logs/RunpSIMS-20130124-0657-6fhtq0vd.log http://www.ci.uchicago.edu/~davidk/logs/cps-2013-01-24_06-56-56.log Thanks, David From wilde at mcs.anl.gov Thu Jan 24 16:36:23 2013 From: wilde at mcs.anl.gov (Michael Wilde) Date: Thu, 24 Jan 2013 16:36:23 -0600 (CST) Subject: [Swift-devel] Why is Swift throttling at 1024 tasks? In-Reply-To: <1348652026.1348480.1359033939697.JavaMail.root@ci.uchicago.edu> Message-ID: <1693567975.1472278.1359066983154.JavaMail.root@mcs.anl.gov> It looks like you specified "foreach.maxthreads=5000" instead of foreach.max.threads=5000 (missing a ".") Obviously the swift command should have complained about this. Can you verify where you specified this property? And test to see if this is indeed the problem? - Mike ----- Original Message ----- > From: "David Kelly" > To: "swift-devel" > Sent: Thursday, January 24, 2013 7:25:39 AM > Subject: [Swift-devel] Why is Swift throttling at 1024 tasks? > > Hello all, > > I am trying to run DSSAT using 4096 cores, but I can't seem to get > Swift to go beyond 1024 active tasks. Looking at the coaster log, I > see all 4096 connecting. I am using persistent coasters, with both > Swift and the coaster service running on the same node. I have my > job throttle set to 4000. I also tried explicitly setting > foreach.max.threads to 4096. I checked ulimit and have a limit of > 4096 open files. Is there some other limit I am missing? > > http://www.ci.uchicago.edu/~davidk/logs/RunpSIMS-20130124-0657-6fhtq0vd.log > http://www.ci.uchicago.edu/~davidk/logs/cps-2013-01-24_06-56-56.log > > Thanks, > David > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > From davidk at ci.uchicago.edu Thu Jan 24 23:08:17 2013 From: davidk at ci.uchicago.edu (David Kelly) Date: Thu, 24 Jan 2013 23:08:17 -0600 (CST) Subject: [Swift-devel] Why is Swift throttling at 1024 tasks? In-Reply-To: <1693567975.1472278.1359066983154.JavaMail.root@mcs.anl.gov> Message-ID: <742459415.1623122.1359090497939.JavaMail.root@ci.uchicago.edu> Ahh, thanks. That was it, I am able use all 4096 cores now. ----- Original Message ----- > From: "Michael Wilde" > To: "David Kelly" > Cc: "swift-devel" > Sent: Thursday, January 24, 2013 4:36:23 PM > Subject: Re: [Swift-devel] Why is Swift throttling at 1024 tasks? > > It looks like you specified "foreach.maxthreads=5000" instead of > foreach.max.threads=5000 (missing a ".") > > Obviously the swift command should have complained about this. > > Can you verify where you specified this property? > > And test to see if this is indeed the problem? > > - Mike > > > > ----- Original Message ----- > > From: "David Kelly" > > To: "swift-devel" > > Sent: Thursday, January 24, 2013 7:25:39 AM > > Subject: [Swift-devel] Why is Swift throttling at 1024 tasks? > > > > Hello all, > > > > I am trying to run DSSAT using 4096 cores, but I can't seem to get > > Swift to go beyond 1024 active tasks. Looking at the coaster log, I > > see all 4096 connecting. I am using persistent coasters, with both > > Swift and the coaster service running on the same node. I have my > > job throttle set to 4000. I also tried explicitly setting > > foreach.max.threads to 4096. I checked ulimit and have a limit of > > 4096 open files. Is there some other limit I am missing? > > > > http://www.ci.uchicago.edu/~davidk/logs/RunpSIMS-20130124-0657-6fhtq0vd.log > > http://www.ci.uchicago.edu/~davidk/logs/cps-2013-01-24_06-56-56.log > > > > Thanks, > > David > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > From iraicu at cs.iit.edu Mon Jan 28 07:51:33 2013 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Mon, 28 Jan 2013 07:51:33 -0600 Subject: [Swift-devel] CFP: IEEE Transactions on Cloud Computing (TCC) Message-ID: <51068265.1030101@cs.iit.edu> Dear Colleagues: The newly established IEEE Transactions on Cloud Computing (TCC) is seeking original and innovative research papers in all areas related to Cloud computing. For details, please see the attached call for papers. Best regards Ioan Raicu Associate Editor IEEE Transactions on Cloud Computing (TCC) ------------------------------------------------------------------------------ IEEE Transactions on Cloud Computing (TCC) ------------------------------------------------------------------------------ Call for Papers IEEE Transactions on Cloud Computing will publish peer-reviewed articles that provide innovative research ideas and applications results in all areas relating to cloud computing. Topics relating to novel theory, algorithms, performance analyses and applications of techniques relating to all areas of cloud computing will be considered for the transactions. The transactions will consider submissions specifically in the areas of cloud security, tradeoffs between privacy and utility of cloud, cloud standards, the architecture of cloud computing, cloud development tools, cloud software, cloud backup and recovery, cloud interoperability, cloud applications management, cloud data analytics, cloud communications protocols, mobile cloud, liability issues for data loss on clouds, data integration on clouds, big data on clouds, cloud education, cloud skill sets, cloud energy consumption, cloud applications in commerce, education and industry. This title will also consider submissions on Infrastructure as a Service (IaaS), Platform as a Service (PaaS), Software as a Service (SaaS), and Business Process as a Service (BPaaS). For details of the submission process, please consult the relevant Web pages at http://www.computer.org/portal/web/tcc ================ Editorial Board: ================ Editor-in-Chief (EiC): ---------------------- Rajkumar Buyya Director, Cloud Computing and Distributed Systems (CLOUDS) Lab The University of Melbourne, Australia CEO, Manjrasoft Pvt Ltd, Melbourne, Australia Web: http://www.cloudbus.org/~raj Associate Editors: ------------------ Adam Wierman, CalTech (California Institute of Technology), USA Albert Zomaya, University of Sydney, Australia Andrew Martin, Oxford University, UK Beng Chin OOI, National University of Singapore, Singapore Beniamino Di Martino, Second University of Naples, Italy Bharadwaj Veeravalli, National University of Singapore, Singapore Carlos Varela, Rensselaer Polytechnic Institute, USA Cesar A. De Rose, PUCRS, Brazil Chandra Krintz, University of California, Santa Barbara, USA Cheng-Zhong Xu, Wayne State University, USA Cho-Li Wang, The University of Hong Kong, China Chunming Rong, University of Stavanger, Norway David Bernstein, Cloud Strategy Partners LLC, USA David De Roure, Oxford University, UK David Lie, University of Toronto, Canada Dejan Milojicic, HP Labs, USA Dick Epema, Delft University of Technology, The Netherlands Gagan Agrawal, Ohio State University, USA Geoffrey Fox, Indiana University, USA Hai Jin, Huazhong University of Science & Technology, China Hui Lei, IBM T. J. Watson Research Center, USA Ignacio Mart?n, Llorente Universidad Complutense de Madrid, Spain Ioan Raicu, Illinois Institute of Technology, Chicago, USA Irena Bojanova, University of Maryland, USA Ivan Stojmenovic, University of Ottawa, Canada Ivona Brandic, Vienna University of Technology, Austria D. Janakiram, Indian Institute of Technology (IIT) Madras, India Jose Fortes, University of Florida, USA Junwei Cao, Tsinghua University, China Kai Hwang, University of Southern California, USA Laurent Lef?vre, INRIA, France Manish Parashar, Rutgers University, USA Masum Z. Hasan, CISCO, USA Murat Demirbas, SUNY Buffalo, USA Omer Rana, Cardiff University, UK Pavan Balaji, Argonne National Laboratory, USA Phillip B. Gibbons, Intel Labs Pittsburgh, USA Pierangela Samarati, Universit? degli Studi di Milano, Italy Qianhui Althea, HP Labs, Singapore Ramesh Sitaraman, University of Massachusetts, USA Rao Kotagiri, University of Melborune, Australia Ricardo Bianchini, Rutgers University, USA Roy Campbell, University of Illinois, Urbana-Champaign, USA Ruby B. Lee, Princeton University, USA Ruppa Thulasiram, University of Manitoba, Canada Sanjeev Aggarwal, Indian Institute of Technology (IIT) Kanpur, India Shivnath Babu, Duke University, USA Shubhashis Sengupta, Accenture, India Siani Pearson, HP Labs, UK Sorav Bansal, Indian Institute of Technology (IIT) Delhi, India Thamarai Selvi, Anna University, India Thomas Fahringer, University of Innsbruck , Austria Umesh Bellur, Indian Institute of Technology (IIT) Mumbai, India Vijaya Varadharajan, Macquarie University, Australia Vojislav Misic, Ryerson University, Canada Xiaofang Zhou, University of Queensland, Australia Yong Cui, Tsinghua University, China ------------------------------------------------------------------------------ -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Editor: IEEE TCC, Springer JoCCASA Chair: IEEE/ACM MTAGS, ACM ScienceCloud, IEEE/ACM DataCloud ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ LinkedIn: http://www.linkedin.com/in/ioanraicu Google: http://scholar.google.com/citations?user=jE73HYAAAAAJ ================================================================= =================================================================