From ketan at mcs.anl.gov Sun Feb 1 19:58:26 2015 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Sun, 1 Feb 2015 19:58:26 -0600 Subject: [Swift-user] Swift runs one task despite slots set to higher number Message-ID: Hi, Running a workflow, I need to run 10 PBS tasks each with a size of 16 nodes. The workflow starts with 4 single task stages and then fans out to a foreach loop with 10 tasks. Here is the sites file I am using: 13400 03:41:00 1 10 2 single 16 16 2.20 10000 /home/ketan/swift.workdir 1 2.06 10000 3 9700 02:39:00 10060 1 1 /home/ketan/swift.workdir However, Swift runs only 1 task throughout the workflow even after reaching the foreach loop. Any suggestions? Thanks, Ketan -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Sun Feb 1 20:04:57 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sun, 1 Feb 2015 18:04:57 -0800 Subject: [Swift-user] Swift runs one task despite slots set to higher number In-Reply-To: References: Message-ID: <1422842697.21697.0.camel@echo> Hi, Can you post the swift log please? Mihael On Sun, 2015-02-01 at 19:58 -0600, Ketan Maheshwari wrote: > Hi, > > Running a workflow, I need to run 10 PBS tasks each with a size of 16 > nodes. > > The workflow starts with 4 single task stages and then fans out to a > foreach loop with 10 tasks. > > Here is the sites file I am using: > > > > > > > 13400 > 03:41:00 > 1 > 10 > 2 > single > 16 > 16 > 2.20 > 10000 > > /home/ketan/swift.workdir > > > > > 1 > 2.06 > 10000 > 3 > 9700 > 02:39:00 > 10060 > 1 > 1 > > > /home/ketan/swift.workdir > > > > > > However, Swift runs only 1 task throughout the workflow even after reaching > the foreach loop. > > Any suggestions? > > Thanks, > Ketan > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user From ketan at mcs.anl.gov Sun Feb 1 20:09:20 2015 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Sun, 1 Feb 2015 20:09:20 -0600 Subject: [Swift-user] Swift runs one task despite slots set to higher number In-Reply-To: <1422842697.21697.0.camel@echo> References: <1422842697.21697.0.camel@echo> Message-ID: Hi Mihael, The run is not yet finished. Please find attached the log so far. Thanks, Ketan On Sun, Feb 1, 2015 at 8:04 PM, Mihael Hategan wrote: > Hi, > > Can you post the swift log please? > > Mihael > > On Sun, 2015-02-01 at 19:58 -0600, Ketan Maheshwari wrote: > > Hi, > > > > Running a workflow, I need to run 10 PBS tasks each with a size of 16 > > nodes. > > > > The workflow starts with 4 single task stages and then fans out to a > > foreach loop with 10 tasks. > > > > Here is the sites file I am using: > > > > > > > > > > > > > > 13400 > > 03:41:00 > > 1 > > 10 > > 2 > > single > > 16 > > 16 > > 2.20 > > 10000 > > > > /home/ketan/swift.workdir > > > > > > > > > > 1 > > 2.06 > > 10000 > > 3 > > 9700 > > 02:39:00 > > 10060 > > 1 > > 1 > > > > > > /home/ketan/swift.workdir > > > > > > > > > > > > However, Swift runs only 1 task throughout the workflow even after > reaching > > the foreach loop. > > > > Any suggestions? > > > > Thanks, > > Ketan > > _______________________________________________ > > Swift-user mailing list > > Swift-user at ci.uchicago.edu > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: run002.tgz Type: application/x-gzip Size: 45547 bytes Desc: not available URL: From hategan at mcs.anl.gov Sun Feb 1 20:57:31 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sun, 1 Feb 2015 18:57:31 -0800 Subject: [Swift-user] Swift runs one task despite slots set to higher number In-Reply-To: References: <1422842697.21697.0.camel@echo> Message-ID: <1422845851.21697.7.camel@echo> You are asking for 16 cores through granularity, but explicitly setting jobType=single. This results PBS only attempting to start one worker for each job. However, the coaster scheduler thinks that 16 workers were requested and, since that is more than the number of queued jobs, it will not start any other ones. Mihael On Sun, 2015-02-01 at 20:09 -0600, Ketan Maheshwari wrote: > Hi Mihael, > > The run is not yet finished. Please find attached the log so far. > > Thanks, > Ketan > > On Sun, Feb 1, 2015 at 8:04 PM, Mihael Hategan wrote: > > > Hi, > > > > Can you post the swift log please? > > > > Mihael > > > > On Sun, 2015-02-01 at 19:58 -0600, Ketan Maheshwari wrote: > > > Hi, > > > > > > Running a workflow, I need to run 10 PBS tasks each with a size of 16 > > > nodes. > > > > > > The workflow starts with 4 single task stages and then fans out to a > > > foreach loop with 10 tasks. > > > > > > Here is the sites file I am using: > > > > > > > > > > > > > > > > > > > > > 13400 > > > 03:41:00 > > > 1 > > > 10 > > > 2 > > > single > > > 16 > > > 16 > > > 2.20 > > > 10000 > > > > > > /home/ketan/swift.workdir > > > > > > > > > > > > > > > 1 > > > 2.06 > > > 10000 > > > 3 > > > 9700 > > > 02:39:00 > > > 10060 > > > 1 > > > 1 > > > > > > > > > /home/ketan/swift.workdir > > > > > > > > > > > > > > > > > > However, Swift runs only 1 task throughout the workflow even after > > reaching > > > the foreach loop. > > > > > > Any suggestions? > > > > > > Thanks, > > > Ketan > > > _______________________________________________ > > > Swift-user mailing list > > > Swift-user at ci.uchicago.edu > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > > > > > _______________________________________________ > > Swift-user mailing list > > Swift-user at ci.uchicago.edu > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > From ketan at mcs.anl.gov Sun Feb 1 21:28:31 2015 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Sun, 1 Feb 2015 21:28:31 -0600 Subject: [Swift-user] Swift runs one task despite slots set to higher number In-Reply-To: <1422845851.21697.7.camel@echo> References: <1422842697.21697.0.camel@echo> <1422845851.21697.7.camel@echo> Message-ID: Hi Mihael, Swift is submitting one 16 *node* job with this setup. My intention is to run 10 such 16 node jobs. Thanks, Ketan On Sun, Feb 1, 2015 at 8:57 PM, Mihael Hategan wrote: > You are asking for 16 cores through granularity, but explicitly setting > jobType=single. This results PBS only attempting to start one worker for > each job. However, the coaster scheduler thinks that 16 workers were > requested and, since that is more than the number of queued jobs, it > will not start any other ones. > > Mihael > > On Sun, 2015-02-01 at 20:09 -0600, Ketan Maheshwari wrote: > > Hi Mihael, > > > > The run is not yet finished. Please find attached the log so far. > > > > Thanks, > > Ketan > > > > On Sun, Feb 1, 2015 at 8:04 PM, Mihael Hategan > wrote: > > > > > Hi, > > > > > > Can you post the swift log please? > > > > > > Mihael > > > > > > On Sun, 2015-02-01 at 19:58 -0600, Ketan Maheshwari wrote: > > > > Hi, > > > > > > > > Running a workflow, I need to run 10 PBS tasks each with a size of 16 > > > > nodes. > > > > > > > > The workflow starts with 4 single task stages and then fans out to a > > > > foreach loop with 10 tasks. > > > > > > > > Here is the sites file I am using: > > > > > > > > > > > > > > > > > > > > > > > > > > > > 13400 > > > > 03:41:00 > > > > 1 > > > > 10 > > > > 2 > > > > single > > > > 16 > > > > 16 > > > > 2.20 > > > > 10000 > > > > > > > > /home/ketan/swift.workdir > > > > > > > > > > > > > > > > > > > > 1 > > > > 2.06 > > > > 10000 > > > > 3 > > > > 9700 > > > > 02:39:00 > > > > 10060 > > > > 1 > > > > 1 > > > > > > > > > > > > /home/ketan/swift.workdir > > > > > > > > > > > > > > > > > > > > > > > > However, Swift runs only 1 task throughout the workflow even after > > > reaching > > > > the foreach loop. > > > > > > > > Any suggestions? > > > > > > > > Thanks, > > > > Ketan > > > > _______________________________________________ > > > > Swift-user mailing list > > > > Swift-user at ci.uchicago.edu > > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > > > > > > > > _______________________________________________ > > > Swift-user mailing list > > > Swift-user at ci.uchicago.edu > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > > > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Sun Feb 1 23:11:04 2015 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sun, 1 Feb 2015 21:11:04 -0800 Subject: [Swift-user] Swift runs one task despite slots set to higher number In-Reply-To: References: <1422842697.21697.0.camel@echo> <1422845851.21697.7.camel@echo> Message-ID: <1422853864.23859.17.camel@echo> On Sun, 2015-02-01 at 21:28 -0600, Ketan Maheshwari wrote: > Hi Mihael, > > Swift is submitting one 16 *node* job with this setup. Maybe, but it thinks it has 16 things that can run jobs, except 15 of them are in limbo because jobType=single forces PBS to only start one worker instance. > My intention is to > run 10 such 16 node jobs. I would suggest trying "single-node" jobs with an appropriately large ppn, since support for MPI jobs is very limited currently. Mihael From karthikeyanb at uchicago.edu Wed Feb 11 11:34:22 2015 From: karthikeyanb at uchicago.edu (Karthikeyan Balasubramanian) Date: Wed, 11 Feb 2015 17:34:22 +0000 Subject: [Swift-user] Optimizing Swift overhead of file copy - reg. Message-ID: <8CEB97C36B499F4CB2FA1E00DD06E343449DCBE0@xm-mbx-07-prod.ad.uchicago.edu> Hi, When jobs are run in Swift, it copies the data from the original location to the work folder for processing. Since our data are in the order of 15-30 GB, it adds unwanted overhead for the process. I recall from my early discussions with Michael Wilde that this could be optimized so that the files can be processed from their original location. I am wondering as how should I proceed to achieve this optimization. Your inputs are greatly appreciated. Thanks. B.K. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yadudoc1729 at gmail.com Wed Feb 11 11:58:08 2015 From: yadudoc1729 at gmail.com (Yadu Nand) Date: Wed, 11 Feb 2015 11:58:08 -0600 Subject: [Swift-user] Optimizing Swift overhead of file copy - reg. In-Reply-To: <8CEB97C36B499F4CB2FA1E00DD06E343449DCBE0@xm-mbx-07-prod.ad.uchicago.edu> References: <8CEB97C36B499F4CB2FA1E00DD06E343449DCBE0@xm-mbx-07-prod.ad.uchicago.edu> Message-ID: Hi Karthikeyan, I'm guessing from the your mention of "processing from their original location" that Mike must have been referring to staging mode direct. In this mode, swift points your applications to your data files which are on a shared filesystem and avoid making copies. There are cases in which application access patterns are not suitable for a shared filesystem, in such cases you can have swift make a copy from the shared-filesystem to a local disk on the node. Since local disks are generally faster than shared-filesystems, this might make more sense if your application is I/O intensive. Here's a document that might help give you a clearer picture : https://github.com/swift-lang/swift-k/blob/master/docs/staging/staging.asc Thanks, Yadu On Wed, Feb 11, 2015 at 11:34 AM, Karthikeyan Balasubramanian < karthikeyanb at uchicago.edu> wrote: > Hi, > > When jobs are run in Swift, it copies the data from the original > location to the work folder for processing. Since our data are in the order > of 15-30 GB, it adds unwanted overhead for the process. I recall from my > early discussions with Michael Wilde that this could be optimized so that > the files can be processed from their original location. > > I am wondering as how should I proceed to achieve this optimization. > Your inputs are greatly appreciated. > > Thanks. > B.K. > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > -- Yadu Nand B -------------- next part -------------- An HTML attachment was scrubbed... URL: From iraicu at cs.iit.edu Fri Feb 13 04:39:14 2015 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Fri, 13 Feb 2015 04:39:14 -0600 Subject: [Swift-user] CFP: IEEE Transaction on Cloud Computing, Special Issue on Many-Task Computing in the Cloud -- Due 03/13/2015 Message-ID: <54DDD452.8030203@cs.iit.edu> Call for Papers --------------------------------------------------------------------------------------- IEEE Transaction on Cloud Computing Special Issue on Many-Task Computing in the Cloud http://datasys.cs.iit.edu/events/TCC-MTC15/ ======================================================================================= The Special Issue on Many-Task Computing (MTC) in the Cloud will provide the scientific community a dedicated forum, within the prestigious IEEE Transactions on Cloud Computing journal, for presenting new research, development, and deployment efforts of loosely coupled large scale applications on Cloud Computing infrastructure. MTC, the theme of this special issue, encompasses loosely coupled applications, which are generally composed of many tasks to achieve some larger application goal. This special issue will cover challenges that can hamper efficiency and utilization in running applications on large-scale systems, such as local resource manager scalability and granularity, efficient utilization of raw hardware, parallel file-system contention and scalability, data management, I/O management, reliability at scale, and application scalability. We welcome paper submissions in theoretical, simulations, and systems topics with special consideration to papers addressing the intersection of petascale/exascale challenges with large-scale cloud computing. We seek submission of papers that present new, original and innovative ideas for the "first" time in TCC (Transactions on Cloud Computing). That means, submission of "extended versions" of already published works (e.g., conference/workshop papers) is not encouraged unless they contain significant number of "new and original" ideas/contributions along with more than 49% brand "new" material. For more information on past workshops and special issues on Many-Task Computing, see http://datasys.cs.iit.edu/events/MTAGS/index.html. We ran a Special Issue on Many-Task Computing in the IEEE Transactions on Parallel and Distributed Systems (TPDS) which appeared in June 2011; the proceedings can be found online at http://www.computer.org/portal/web/csdl/abs/trans/td/2011/06/ttd201106toc.htm. The special issue editors also published a highly relevant paper that defines Many-Task Computing which was published in the inaugural MTAGS08 workshop, titled "Many-Task Computing for Grids and Supercomputers"; we encourage potential authors to read this paper, and to clearly articulate in your paper submissions how your papers are related to Many-Task Computing. For more information on this special issue, please see http://datasys.cs.iit.edu/events/TCC-MTC15/. Topics --------------------------------------------------------------------------------------- We seek submission of papers that present new, original and innovative ideas for the "first" time in TCC (Transactions on Cloud Computing). That means, submission of "extended versions" of already published works (e.g., conference/workshop papers) will only be encouraged if they contain significant number of "new and original" ideas/contributions along with more than 49% brand "new" material. TCC expects submissions to be complete in all respects including author names, affiliation, bios etc. Manuscript should be 14 double column pages (all regular paper page limits include references and author biographies). We aim to cover topics related to Many-Task Computing and Cloud Computing. Topics of interest include: * Compute Resource Management * Scheduling * Job execution frameworks * Local resource manager extensions * Performance evaluation of resource managers in use on large scale systems * Dynamic resource provisioning * Techniques to manage many-core resources and/or GPUs * Challenges and opportunities in running many-task workloads on Cloud Computing infrastructure * Storage architectures and implementations * Distributed file systems * Parallel file systems * Distributed meta-data management * Content distribution systems for large data * Data caching frameworks and techniques * Data management within and across data centers * Data-aware scheduling * Data-intensive computing applications * Eventual-consistency storage usage and management * Programming models and tools * Map-reduce and its generalizations * Many-task computing middleware and applications * Parallel programming frameworks * Ensemble MPI techniques and frameworks * Service-oriented science applications * Large-Scale Workflow Systems * Workflow system performance and scalability analysis * Scalability of workflow systems * Workflow infrastructure and e-Science middleware * Programming Paradigms and Models * Large-Scale Many-Task Applications * High-throughput computing (HTC) applications * Data-intensive applications * Quasi-supercomputing applications, deployments, and experiences * Performance Evaluation * Performance evaluation * Real systems * Simulations * Reliability of large systems Paper Submission and Publication --------------------------------------------------------------------------------------- Authors are invited to submit unpublished and original work to the IEEE Transactions on Cloud Computing (TCC), Special Issue on Many-Task Computing in the Cloud. If the paper is extended from an initial work, the submission must contain at least 50% new material that can be qualified as ?brand? new ideas and results. The paper must be in the IEEE TCC format, namely 14 double-column pages or 30 single-column pages (Note: All regular paper page limits include references and author biographies). Please note that the double-column format will translate more readily into the final publication format. A double-column page is defined as a 7.875??10.75? page with 10-point type, 12-point vertical spacing, and 0.5 inch margins. A single-column page is defined as an 8.5??11? page with 12-point type and 24-point vertical spacing, containing approximately 250 words. All of the margins should be one inch (top, bottom, right and left). These length limits are taking into account reasonably-sized figures and references. Papers must be submitted using the submission system: https://mc.manuscriptcentral.com/tcc-cs, by selecting the special issue option ?SI-MTC?. For more information, please see http://datasys.cs.iit.edu/events/TCC-MTC15/. Email can be sent to the editors with further questions at mtc15-tcc-editors at datasys.cs.iit.edu. Important Dates --------------------------------------------------------------------------------------- * Papers Due: March 13th, 2015 * First round decisions: June 1st, 2015 * Major Revisions if needed: August 1st, 2015 * Final decisions: September 1st, 2015 * Publication Date: Fall 2015 (may vary depending on production queue) Guest Editors (mtc15-tcc-editors at datasys.cs.iit.edu) --------------------------------------------------------------------------------------- * Ioan Raicu, Illinois Institute of Technology & Argonne National Laboratory * Justin Wozniak, University of Chicago & Argonne National Laboratory * Ian Foster, University of Chicago & Argonne National Laboratory * Yong Zhao, University of Electronic Science and Technology of China From iraicu at cs.iit.edu Sun Feb 15 08:14:57 2015 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Sun, 15 Feb 2015 08:14:57 -0600 Subject: [Swift-user] CFP: IEEE Cluster 2015 -- Papers due 02/27/15 Message-ID: <54E0A9E1.80501@cs.iit.edu> IEEE International Conference on Cluster Computing September 8-11, 2015 Chicago, IL, USA http://www.mcs.anl.gov/ieeecluster2015/ ---------------------------------------------- ...Follow us on Facebook athttps://www.facebook.com/ieee.cluster ...Follow us on Twitter athttps://twitter.com/IEEECluster ...Follow us on Linkedin at https://www.linkedin.com/groups/IEEE-International-Conference-on-Cluster-7428925 ...Follow us on RenRen athttp://page.renren.com/601871401 ---------------------------------------------- CALL FOR PAPERS Following the successes of the series of Cluster conferences, for 2015 we solicit high-quality original papers presenting work that advances the state-of-the-art in clusters and closely related fields. All papers will be rigorously peer-reviewed for their originality, technical depth and correctness, potential impact, relevance to the conference, and quality of presentation. Research papers must clearly demonstrate research contributions and novelty, while papers reporting experience must clearly describe lessons learned and impact, along with the utility of the approach compared to the ones in the past. PAPER TRACKS * Applications, Algorithms, and Libraries * Architecture, Networks/Communication, and Management * Programming and Systems Software * Data, Storage, and Visualization SUBMISSION GUIDELINES Authors are invited to submit papers electronically in PDF format. Submitted manuscripts should be structured as technical papers and may not exceed 10 letter-size (8.5 x 11) pages including figures, tables and references using the IEEE format for conference proceedings. Submissions not conforming to these guidelines may be returned without review. Authors should make sure that their file will print on a printer that uses letter-size (8.5 x 11) paper. The official language of the conference is English. All manuscripts will be reviewed and will be judged on correctness, originality, technical strength, significance, quality of presentation, and interest and relevance to the conference attendees. Paper submissions are limited to 10 pages in 2-column IEEE format including all figures and references. Submitted manuscripts exceeding this limit will be returned without review. For the final camera-ready version, authors with accepted papers may purchase additional pages at the following rates: 200 USD for each of two additional pages. See formatting templates for details: * LaTex Package http://datasys.cs.iit.edu/events/CCGrid2014/IEEECS_confs_LaTeX.zip * Word Template http://datasys.cs.iit.edu/events/CCGrid2014/instruct8.5x11x2.doc and http://datasys.cs.iit.edu/events/CCGrid2014/instruct8.5x11x2.pdf Submitted papers must represent original unpublished research that is not currently under review for any other conference or journal. Papers not following these guidelines will be rejected without review and further action may be taken, including (but not limited to) notifications sent to the heads of the institutions of the authors and sponsors of the conference. Submissions received after the due date, exceeding the page limit, or not appropriately structured may not be considered. Authors may contact the conference chairs for more information. The proceedings will be published through the IEEE Computer Society Conference Publishing Services. Please submit your paper via the EasyChair submission system: https://easychair.org/conferences/?conf=ieeecluster2015 JOURNAL SPECIAL ISSUE The best papers of Cluster 2015 will be included in a Special Issue on advances in topics related to cluster computing of the Elsevier International Journal of Parallel Computing (PARCO), edited by Pavan Balaji, Satoshi Matsuoka, and Michela Taufer. This special issue is dedicated for the papers accepted in the Cluster 2015 conference. The submission to this special issue is by invitation only. IMPORTANT DATES January 1, 2015 ........... Submissions open for Papers February 27, 2015 ....... Papers Submission Deadline April 23, 2015 ............... Papers Acceptance Notification August 1, 2015 ............ Camera-ready Copy Deadline for Papers See other important dates here http://www.mcs.anl.gov/ieeecluster2015/author-information/important-dates/. CLUSTER 2015 PROGRAM CHAIR Satoshi Matsuoka, Tokyo Institute of Technology (matsu AT is.titech.ac.jp). -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Editor: IEEE TCC, Springer Cluster, Springer JoCCASA Chair: IEEE/ACM MTAGS, ACM ScienceCloud ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ LinkedIn: http://www.linkedin.com/in/ioanraicu Google: http://scholar.google.com/citations?user=jE73HYAAAAAJ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From kdontula at hawk.iit.edu Wed Feb 11 14:03:34 2015 From: kdontula at hawk.iit.edu (Kaushal Dontula) Date: Wed, 11 Feb 2015 20:03:34 -0000 Subject: [Swift-user] Error while compiling Swift program Message-ID: Hello, Can you please help me with the following error Execution failed: The application "bash" is not available for any site/pool in your tc.data catalog splitFile, wordcount.swift, line 12 Thanks in advance. Regards, Kaushal. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yadudoc1729 at gmail.com Tue Feb 17 15:02:53 2015 From: yadudoc1729 at gmail.com (Yadu Nand) Date: Tue, 17 Feb 2015 15:02:53 -0600 Subject: [Swift-user] Error while compiling Swift program In-Reply-To: References: Message-ID: Hi Kaushal, Could you send us logs from your run ? If you running a swift-0.95-x version, the runNNN folders generated are what we need. Thanks, Yadu On Wed, Dec 10, 2014 at 9:35 PM, Kaushal Dontula wrote: > Hello, > > Can you please help me with the following error > > Execution failed: > The application "bash" is not available for any site/pool in your tc.data catalog > splitFile, wordcount.swift, line 12 > > Thanks in advance. > > Regards, > > Kaushal. > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > -- Yadu Nand B -------------- next part -------------- An HTML attachment was scrubbed... URL: From iraicu at cs.iit.edu Mon Feb 23 10:31:38 2015 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Mon, 23 Feb 2015 11:31:38 -0500 Subject: [Swift-user] CFP: IEEE CLUSTER 2015 -- deadline extension March 7, 2015 Message-ID: <54EB55EA.2000909@cs.iit.edu> February 23, 2015 Release IEEE International Conference on Cluster Computing September 8-11, 2015 Chicago, IL, USA http://www.mcs.anl.gov/ieeecluster2015/ CLUSTER 2015 CALL FOR PAPERS Following the successes of the series of Cluster conferences, for 2015 we solicit high-quality original papers presenting work that advances the state-of-the-art in clusters and closely related fields. All papers will be rigorously peer-reviewed for their originality, technical depth and correctness, potential impact, relevance to the conference, and quality of presentation. Research papers must clearly demonstrate research contributions and novelty, while papers reporting experience must clearly describe lessons learned and impact, along with the utility of the approach compared to the ones in the past. *** Paper Tracks *** Applications, Algorithms, and Libraries Architecture, Networks/Communication, and Management Programming and Systems Software Data, Storage, and Visualization *** Submission Guidelines *** Authors are invited to submit papers electronically in PDF format. Submitted manuscripts should be structured as technical papers and may not exceed 10 letter-size (8.5 x 11) pages including figures, tables and references using the IEEE format for conference proceedings. Submissions not conforming to these guidelines may be returned without review. Authors should make sure that their file will print on a printer that uses letter-size (8.5 x 11) paper. The official language of the conference is English. All manuscripts will be reviewed and will be judged on correctness, originality, technical strength, significance, quality of presentation, and interest and relevance to the conference attendees. Paper submissions are limited to 10 pages in 2-column IEEE format including all figures and references. Submitted manuscripts exceeding this limit will be returned without review. For the final camera-ready version, authors with accepted papers may purchase additional pages at the following rates: 200 USD for each of two additional pages. See formatting templates for details: LaTex Package ZIP (http://datasys.cs.iit.edu/events/CCGrid2014/IEEECS_confs_LaTeX.zip) Word Template DOC (http://datasys.cs.iit.edu/events/CCGrid2014/instruct8.5x11x2.doc) and PDF (http://datasys.cs.iit.edu/events/CCGrid2014/instruct8.5x11x2.pdf) Submitted papers must represent original unpublished research that is not currently under review for any other conference or journal. Papers not following these guidelines will be rejected without review and further action may be taken, including (but not limited to) notifications sent to the heads of the institutions of the authors and sponsors of the conference. Submissions received after the due date, exceeding the page limit, or not appropriately structured may not be considered. Authors may contact the conference chairs for more information. The proceedings will be published through the IEEE Computer Society Conference Publishing Services. Please submit your paper via the EasyChair submission system: https://easychair.org/conferences/?conf=ieeecluster2015 *** Journal Special Issue *** The best papers of Cluster 2015 will be included in a Special Issue on advances in topics related to cluster computing of the Elsevier International Journal of Parallel Computing (PARCO), edited by Pavan Balaji, Satoshi Matsuoka, and Michela Taufer. This special issue is dedicated for the papers accepted in the Cluster 2015 conference. The submission to this special issue is by invitation only. *** Important Dates *** ***March 7, 2015*** Papers Submission Deadline April 23, 2015 Papers Acceptance Notification See other deadlines in the Important Dates page (http://www.mcs.anl.gov/ieeecluster2015/author-information/important-dates) *** Cluster 2015 Program Chair *** Satoshi Matsuoka, Tokyo Institute of Technology (matsu AT is.titech.ac.jp). ---------------------------------------------- ...Follow us on Facebook athttps://www.facebook.com/ieee.cluster ...Follow us on Twitter athttps://twitter.com/IEEECluster ...Follow us on Linkedin at https://www.linkedin.com/groups/IEEE-International-Conference-on-Cluster-7428925 ...Follow us on RenRen athttp://page.renren.com/601871401 ---------------------------------------------- -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Editor: IEEE TCC, Springer Cluster, Springer JoCCASA Chair: IEEE/ACM MTAGS, ACM ScienceCloud ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ LinkedIn: http://www.linkedin.com/in/ioanraicu Google: http://scholar.google.com/citations?user=jE73HYAAAAAJ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From Matthew.Shaxted at som.com Wed Feb 25 10:12:52 2015 From: Matthew.Shaxted at som.com (Matthew Shaxted) Date: Wed, 25 Feb 2015 11:12:52 -0500 Subject: [Swift-user] Swift/MPI & OpenFoam Message-ID: Hi all, A few questions about Swift's interaction with MPI - specifically related to OpenFoam CFD: To date, we are running OpenFoam CFD jobs on a single machine at the moment, but are looking to scale up computation by way of OpenFoam's MPI support. We discussed a strategy of enabling Lustre or a similar file system across our internal resources, connecting our worker nodes to the head node via an MPI listener, and submitting/running the jobs via OpenFoam's MPI run command. Does Swift support a connection via MPI? Perhaps Swift can help manage setting up the in the same way a worker.pl is setup for coaster-service? Thanks for any insights. Matthew MATTHEW SHAXTED SKIDMORE, OWINGS & MERRILL LLP 224 SOUTH MICHIGAN AVENUE CHICAGO, IL 60604 T (312) 360-4368 MATTHEW.SHAXTED at SOM.COM [cid:image004.png at 01D050E3.9D61B950] The information contained in this communication may be confidential, is intended only for the use of the recipient(s) named above, and may be legally privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited and may be unlawful. If you have received this communication in error, please return it to the sender immediately and delete the original message and any copy of it from your computer system. If you have any questions concerning this message, please contact the sender. [cid:image003.gif at 01D050E2.BD72C230] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.gif Type: image/gif Size: 566 bytes Desc: image003.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.png Type: image/png Size: 5311 bytes Desc: image004.png URL: From wozniak at mcs.anl.gov Wed Feb 25 09:51:27 2015 From: wozniak at mcs.anl.gov (Justin M Wozniak) Date: Wed, 25 Feb 2015 09:51:27 -0600 Subject: [Swift-user] Swift/MPI & OpenFoam In-Reply-To: References: Message-ID: <54EDEF7F.4070201@mcs.anl.gov> Swift/T has a feature to call an MPI library: http://swift-lang.org/Swift-T/leaf.html#_complete_example_8_calling_an_mpi_library Does that seem appropriate? On 02/25/2015 10:12 AM, Matthew Shaxted wrote: > > Hi all, > > A few questions about Swift?s interaction with MPI ? specifically > related to OpenFoam CFD: > > To date, we are running OpenFoam CFD jobs on a single machine at the > moment, but are looking to scale up computation by way of OpenFoam?s > MPI support. > > We discussed a strategy of enabling Lustre or a similar file system > across our internal resources, connecting our worker nodes to the head > node via an MPI listener, and submitting/running the jobs via > OpenFoam?s MPI run command. > > Does Swift support a connection via MPI? > > Perhaps Swift can help manage setting up the in the same way a > worker.pl is setup for coaster-service? > > Thanks for any insights. > > Matthew > > MATTHEW SHAXTED > > SKIDMORE, OWINGS & MERRILL LLP > 224 SOUTH MICHIGAN AVENUE > CHICAGO, IL 60604 > T (312) 360-4368 > MATTHEW.SHAXTED at SOM.COM > > cid:image001.png at 01CF9071.6FB46030 > > The information contained in this communication may be confidential, > is intended only for the use of the recipient(s) named above, and may > be legally privileged. If the reader of this message is not the > intended recipient, you are hereby notified that any dissemination, > distribution, or copying of this communication, or any of its > contents, is strictly prohibited and may be unlawful. If you have > received this communication in error, please return it to the > sen???der immediately and delete the original message and any copy of > it from your computer system. If you have any questions concerning > this message, please contact the sender. > > http://intranet.som.com/common/admin/file.cfm?f=%2Fresources%2Fcontent%2F5%2F0%2F4%2F4%2F6%2F4%2F0%2F3%2Fdocuments%2Fimagea560bf%2Egif%406e10073b%2E30854c37 > > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user -- Justin M Wozniak -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 5311 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 566 bytes Desc: not available URL: From yadunand at uchicago.edu Wed Feb 25 14:08:35 2015 From: yadunand at uchicago.edu (Yadu Nand Babuji) Date: Wed, 25 Feb 2015 14:08:35 -0600 Subject: [Swift-user] Swift/MPI & OpenFoam In-Reply-To: References: Message-ID: <54EE2BC3.4040008@uchicago.edu> Hi Matthew, I am in the middle of updating our MPI documentation, and I think it would be best to address this question through a generalized example. Here's what I have in mind : You have hosts, node001...008, that you have password-less ssh access to. 1. Set hosts to node001...node008, and hostcount to N. 2. A script similar to start-coaster-service accesses the above variables and creates partitions with hostcount number of hosts. The script then starts the swift worker on the first node in each partition of hosts, and sets a node list environment variable. Eg. you have node001 ... node008 and hostcount = 4 Each worker(of 2) gets a portion of the nodes, the worker on node001 gets NODE_LIST=[node001 node002 node003 node4] 3. Tasks submitted to the worker can use mpirun --host to start mpi tasks with hostcount number of nodes. 4. Run mpi tasks to check if the ranks are on the right nodes. Does this sound reasonable ? Thanks, Yadu On 02/25/2015 10:12 AM, Matthew Shaxted wrote: > > Hi all, > > A few questions about Swift?s interaction with MPI ? specifically > related to OpenFoam CFD: > > To date, we are running OpenFoam CFD jobs on a single machine at the > moment, but are looking to scale up computation by way of OpenFoam?s > MPI support. > > We discussed a strategy of enabling Lustre or a similar file system > across our internal resources, connecting our worker nodes to the head > node via an MPI listener, and submitting/running the jobs via > OpenFoam?s MPI run command. > > Does Swift support a connection via MPI? > > Perhaps Swift can help manage setting up the in the same way a > worker.pl is setup for coaster-service? > > Thanks for any insights. > > Matthew > > MATTHEW SHAXTED > > SKIDMORE, OWINGS & MERRILL LLP > 224 SOUTH MICHIGAN AVENUE > CHICAGO, IL 60604 > T (312) 360-4368 > MATTHEW.SHAXTED at SOM.COM > > cid:image001.png at 01CF9071.6FB46030 > > The information contained in this communication may be confidential, > is intended only for the use of the recipient(s) named above, and may > be legally privileged. If the reader of this message is not the > intended recipient, you are hereby notified that any dissemination, > distribution, or copying of this communication, or any of its > contents, is strictly prohibited and may be unlawful. If you have > received this communication in error, please return it to the > sen???der immediately and delete the original message and any copy of > it from your computer system. If you have any questions concerning > this message, please contact the sender. > > http://intranet.som.com/common/admin/file.cfm?f=%2Fresources%2Fcontent%2F5%2F0%2F4%2F4%2F6%2F4%2F0%2F3%2Fdocuments%2Fimagea560bf%2Egif%406e10073b%2E30854c37 > > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 5311 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 566 bytes Desc: not available URL: