From lgadelha at lncc.br Sun Oct 11 17:34:16 2009 From: lgadelha at lncc.br (Luiz M. R. Gadelha Jr.) Date: Sun, 11 Oct 2009 19:34:16 -0300 (BRT) Subject: [Swift-devel] PC3 technical report Message-ID: <60543.189.60.67.134.1255300456.squirrel@webmail.lncc.br> Hi, I'm sending a technical report describing our participation in the Third Provenance Challenge (PC3). It is also intended to serve as a tutorial for those interested testing Swift's provenance features. Any suggestion and contributions are welcome. There's an upcoming PC3 special edition of FGCS: http://www.cs.wisc.edu/dbworld/messages/2009-07/1248118814.html Regards, Luiz -- Luiz M. R. Gadelha Jr. - LNCC http://www.lncc.br/~lgadelha -------------- next part -------------- A non-text attachment was scrubbed... Name: pc3_tr.odt Type: application/vnd.oasis.opendocument.text Size: 157019 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pc3_tr.doc Type: application/msword Size: 342016 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pc3_tr.pdf Type: application/pdf Size: 164391 bytes Desc: not available URL: From skenny at uchicago.edu Mon Oct 12 00:35:58 2009 From: skenny at uchicago.edu (skenny at uchicago.edu) Date: Mon, 12 Oct 2009 00:35:58 -0500 (CDT) Subject: [Swift-devel] script for extracting compute time Message-ID: <20091012003558.CDS23038@m4500-02.uchicago.edu> a quick and dirty python script for getting the total compute time for a workflow from a swift log file: https://trac.ci.uchicago.edu/cnari/browser/scripts/logstat.py don't know if others need such a thing, but thought it was worth posting just in case :) personally, i have so many log files at this point i think i'm actually going to use swift to process my swift logs, heh ~sk From hategan at mcs.anl.gov Mon Oct 12 01:10:29 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 12 Oct 2009 01:10:29 -0500 Subject: [Swift-devel] script for extracting compute time In-Reply-To: <20091012003558.CDS23038@m4500-02.uchicago.edu> References: <20091012003558.CDS23038@m4500-02.uchicago.edu> Message-ID: <1255327829.9286.2.camel@localhost> On Mon, 2009-10-12 at 00:35 -0500, skenny at uchicago.edu wrote: > a quick and dirty python script for getting the total compute > time for a workflow from a swift log file: > > https://trac.ci.uchicago.edu/cnari/browser/scripts/logstat.py > > don't know if others need such a thing, but thought it was > worth posting just in case :) I think it's a very good idea and it should be sitting in swift/bin > > personally, i have so many log files at this point i think i'm > actually going to use swift to process my swift logs, heh :) From skenny at uchicago.edu Tue Oct 13 11:14:17 2009 From: skenny at uchicago.edu (skenny at uchicago.edu) Date: Tue, 13 Oct 2009 11:14:17 -0500 (CDT) Subject: [Swift-devel] burnin' up ranger w/the latest coasters Message-ID: <20091013111417.CDU59058@m4500-02.uchicago.edu> Final status: Finished successfully:131072 re-running some of the workflows from our recent SEM paper with the latest swift...sadly, queue time on ranger has only gone up since those initial runs...but luckily coasters has speeded things up, so it ends up evening out for time to solution :) not sure i fully understand the plot: http://www.ci.uchicago.edu/~skenny/workflows/sem_131k/ log is here: /ci/projects/cnari/logs/skenny/4reg_2cond-20091012-1607-ugidm2s2.log From foster at anl.gov Tue Oct 13 11:23:21 2009 From: foster at anl.gov (Ian Foster) Date: Tue, 13 Oct 2009 11:23:21 -0500 Subject: [Swift-devel] burnin' up ranger w/the latest coasters In-Reply-To: <20091013111417.CDU59058@m4500-02.uchicago.edu> References: <20091013111417.CDU59058@m4500-02.uchicago.edu> Message-ID: <075C230D-8AA8-467F-A07C-057A589EF762@anl.gov> Sarah: This looks nice. It would be nice to have a plot showing Coaster activity, as well. (Jobs submitted, coasters allocated, etc.) Mihael, Mike: any chance of that? Ian. On Oct 13, 2009, at 11:14 AM, wrote: > Final status: Finished successfully:131072 > > re-running some of the workflows from our recent SEM > paper with the latest swift...sadly, queue time on ranger has > only gone up since those initial runs...but luckily coasters > has speeded things up, so it ends up evening out for time to > solution :) > > not sure i fully understand the plot: > > http://www.ci.uchicago.edu/~skenny/workflows/sem_131k/ > > log is here: > > /ci/projects/cnari/logs/skenny/4reg_2cond-20091012-1607-ugidm2s2.log > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel From hategan at mcs.anl.gov Tue Oct 13 11:27:11 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 13 Oct 2009 11:27:11 -0500 Subject: [Swift-devel] burnin' up ranger w/the latest coasters In-Reply-To: <075C230D-8AA8-467F-A07C-057A589EF762@anl.gov> References: <20091013111417.CDU59058@m4500-02.uchicago.edu> <075C230D-8AA8-467F-A07C-057A589EF762@anl.gov> Message-ID: <1255451231.12836.1.camel@localhost> On Tue, 2009-10-13 at 11:23 -0500, Ian Foster wrote: > Sarah: > > This looks nice. It would be nice to have a plot showing Coaster > activity, as well. (Jobs submitted, coasters allocated, etc.) Mihael, > Mike: any chance of that? That would be useful. I'll see what I can do. From iraicu at cs.uchicago.edu Tue Oct 13 17:15:19 2009 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Tue, 13 Oct 2009 17:15:19 -0500 Subject: [Swift-devel] CFP: ACM Int. Symposium High Performance Distributed Computing (HPDC) 2010 Message-ID: <4AD4FBF7.4070601@cs.uchicago.edu> ACM HPDC 2010 Call For Papers 19th ACM International Symposium on High Performance Distributed Computing Chicago, Illinois June 21-25, 2010 http://hpdc2010.eecs.northwestern.edu The ACM International Symposium on High Performance Distributed Computing (HPDC) is the premier venue for presenting the latest research on the design, implementation, evaluation, and use of parallel and distributed systems for high performance and high end computing. The 19th installment of HPDC will take place in the heart of the Chicago, Illinois, the third largest city in the United States and a major technological and cultural capital. The conference will be held on June 23-25 (Wednesday through Friday) with affiliated workshops occurring on June 21-22 (Monday and Tuesday). Submissions are welcomed on all forms of high performance distributed computing, including grids, clouds, clusters, service-oriented computing, utility computing, peer-to-peer systems, and global computing ensembles. New scholarly research showing empirical and reproducible results in architectures, systems, and networks is strongly encouraged, as are experience reports of applications and deployments that can provide insights for future high performance distributed computing research. All papers will be rigorously reviewed by a distinguished program committee, with a strong focus on the combination of rigorous scientific results and likely high impact within high performance distributed computing. Research papers must clearly demonstrate research contributions and novelty while experience reports must clearly describe lessons learned and demonstrate impact. Topics of interest include (but are not limited to) the following, in the context of high performance distributed computing and high end computing: * Systems * Architectures * Algorithms * Networking * Programming languages and environments * Data management * I/O and file systems * Virtualization * Resource management, scheduling, and load-balancing * Performance modeling, simulation, and prediction * Fault tolerance, reliability and availability * Security, configuration, policy, and management issues * Multicore issues and opportunities * Models and use cases for utility, grid, and cloud computing Both full papers and short papers (for poster presentation and/or demonstrations) may be submitted. IMPORTANT DATES Paper Abstract submissions: January 15, 2010 Paper submissions: January 22, 2010 Author notification: March 30, 2010 Final manuscripts: April 23, 2010 SUBMISSIONS Authors are invited to submit full papers of at most 12 pages or short papers of at most 4 pages. The page limits include all figures and references. Papers should be formatted in the ACM proceedings style (e.g., http://www.acm.org/sigs/publications/proceedings-templates). Reviewing is single-blind. Papers must be self-contained and provide the technical substance required for the program committee to evaluate the paper's contribution, including how it differs from prior work. All papers will be reviewed and judged on correctness, originality, technical strength, significance, quality of presentation, and interest and relevance to the conference. Submitted papers must be original work that has not appeared in and is not under consideration for another conference or a journal. There will be NO DEADLINE EXTENSIONS. PUBLICATION Accepted full and short papers will appear in the conference proceedings. WORKSHOPS A separate call for workshops is available at http://hpdc2010.eecs.northwestern.edu/hpdc2010-cfw.txt. The deadline for workshop proposals is November 2, 2009. GENERAL CO-CHAIRS Kate Keahey, Argonne National Labs Salim Hariri, University of Arizona STEERING COMMITTEE Salim Hariri, Univ. of Arizona (Chair) Andrew A. Chien, Intel / UCSD Henri Bal, Vrije University Franck Cappello, INRIA Jack Dongarra, Univ. of Tennessee Ian Foster, ANL& Univ. of Chicago Andrew Grimshaw, Univ. of Virginia Carl Kesselman, USC/ISI Dieter Kranzlmueller, Ludwig-Maximilians-Univ. Muenchen Miron Livny, Univ. of Wisconsin Manish Parashar, Rutgers University Karsten Schwan, Georgia Tech David Walker, Univ. of Cardiff Rich Wolski, UCSB PROGRAM CHAIR Peter Dinda, Northwestern University PROGRAM COMMITTEE Ron Brightwell, Sandia National Labs Fabian Bustamante, Northwestern University Henri Bal, Vrije Universiteit Frank Cappello, INRIA Claris Castillo, IBM Research Henri Casanova, University of Hawaii Abhishek Chandra, University of Minnesota Chris Colohan, Google Brian Cooper, Yahoo Research Wu-chun Feng, Virginia Tech Jose Fortes, University of Florida Ian Foster, University of Chicago / Argonne Geoffrey Fox, Indiana University Michael Gerndt, TU-Munich Andrew Grimshaw, University of Virginia Thilo Kielmann, Vrije Universiteit Arthur Maccabe, Oak Ridge National Labs Satoshi Matsuoka, Toyota Institute of Technology Jose Moreira, IBM Research Klara Nahrstedt, UIUC Dushyanth Narayanan, Microsoft Research Manish Parashar, Rutgers University Joel Saltz, Emory University Karsten Schwan, Georgia Tech Thomas Stricker, Google Jaspal Subhlok, University of Houston Michela Taufer, University of Delaware Valerie Taylor, TAMU Douglas Thain, University of Notre Dame Jon Weissman, University of Minnesota Rich Wolski, UCSB and Eucalyptus Systems Dongyan Xu, Purdue University Ken Yocum, UCSD WORKSHOP CHAIR Douglas Thain, University of Notre Dame PUBLICITY CO-CHAIRS Martin Swany, U. Delaware Morris Riedel, Juelich Supercomputing Centre Renato Ferreira, Universidade Federal de Minas Gerais Kento Aida, NII and Tokyo Institute of Technology LOCAL ARRANGEMENTS CHAIR Zhiling Lan, IIT STUDENT ACTIVITIES CO-CHAIRS John Lange, Northwestern University Ioan Raicu, Northwestern University -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ http://cucis.ece.northwestern.edu/ ================================================================= ================================================================= From skenny at uchicago.edu Wed Oct 14 15:39:15 2009 From: skenny at uchicago.edu (skenny at uchicago.edu) Date: Wed, 14 Oct 2009 15:39:15 -0500 (CDT) Subject: [Swift-devel] burnin' up ranger w/the latest coasters In-Reply-To: <20091013111417.CDU59058@m4500-02.uchicago.edu> References: <20091013111417.CDU59058@m4500-02.uchicago.edu> Message-ID: <20091014153915.CDW69329@m4500-02.uchicago.edu> for those interested, here are the config files used for this run: swift.properties: sites.file=config/coaster_ranger.xml tc.file=/ci/projects/cnari/config/tc.data lazy.errors=false caching.algorithm=LRU pgraph=false pgraph.graph.options=splines="compound", rankdir="TB" pgraph.node.options=color="seagreen", style="filled" clustering.enabled=false clustering.queue.delay=4 clustering.min.time=60 kickstart.enabled=maybe kickstart.always.transfer=false wrapperlog.always.transfer=false throttle.submit=3 throttle.host.submit=8 throttle.score.job.factor=64 throttle.transfers=16 throttle.file.operations=16 sitedir.keep=false execution.retries=3 replication.enabled=false replication.min.queue.time=60 replication.limit=3 foreach.max.threads=16384 coaster_ranger.xml: 1000.0 normal 32 1 16 8192 72000 TG-DBS080004N /work/00926/tg459516/sidgrid_out/{username} ---- Original message ---- >Date: Tue, 13 Oct 2009 11:14:17 -0500 (CDT) >From: >Subject: [Swift-devel] burnin' up ranger w/the latest coasters >To: swift-devel at ci.uchicago.edu > >Final status: Finished successfully:131072 > >re-running some of the workflows from our recent SEM >paper with the latest swift...sadly, queue time on ranger has >only gone up since those initial runs...but luckily coasters >has speeded things up, so it ends up evening out for time to >solution :) > >not sure i fully understand the plot: > >http://www.ci.uchicago.edu/~skenny/workflows/sem_131k/ > >log is here: > >/ci/projects/cnari/logs/skenny/4reg_2cond-20091012-1607-ugidm2s2.log >_______________________________________________ >Swift-devel mailing list >Swift-devel at ci.uchicago.edu >http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel From iraicu at cs.uchicago.edu Wed Oct 14 16:41:18 2009 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Wed, 14 Oct 2009 16:41:18 -0500 Subject: [Swift-devel] Call for Workshops: ACM Int. Symposium High Performance Distributed Computing (HPDC) 2010 Message-ID: <4AD6457E.8040004@cs.uchicago.edu> HPDC 2010 - Call for Workshops http://hpdc2010.eecs.northwestern.edu We invite proposals for workshops to be held with the ACM Symposium on High Performance Distributed Computing to be held in Chicago, Illinois in June 2010. Workshops will be held June 21-22, preceding the main conference sessions June 23-25. Workshops provide forums for discussion among researchers and practitioners on focused topics or emerging research areas. Workshops may be organized in whatever way is appropriate to the topic, possibly including invited talks, panel discussions, presentation of work in progress, or full peer-reviewer papers. Each workshop will be a full day event hosting 20-40 participants. A workshop must be proposed in writing and sent to dthain at nd.edu. A workshop proposal should consist of: * Name of the workshop. * A few paragraphs describing the theme of the workshop and how it relates to the overall conference. * Data about previous offerings of the workshop, including attendance, number of papers or presentations submitted and accepted. * Names and affiliations of the workshop organizers, and if applicable, a significant portion of the program committee. * Plan for attracting submissions and attendees. * Timeline for milestones such as call for papers, submission deadline, and so forth. Workshop Proposal Deadline: November 2, 2009 Workshop Notification: November 9, 2009 Workshop Calls Online: November 23, 2009 Workshop Proceedings Due: April 23, 2010 -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= From hategan at mcs.anl.gov Thu Oct 15 19:21:40 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 15 Oct 2009 19:21:40 -0500 Subject: [Swift-devel] burnin' up ranger w/the latest coasters In-Reply-To: <1255451231.12836.1.camel@localhost> References: <20091013111417.CDU59058@m4500-02.uchicago.edu> <075C230D-8AA8-467F-A07C-057A589EF762@anl.gov> <1255451231.12836.1.camel@localhost> Message-ID: <1255652500.9269.1.camel@localhost> On Tue, 2009-10-13 at 11:27 -0500, Mihael Hategan wrote: > On Tue, 2009-10-13 at 11:23 -0500, Ian Foster wrote: > > Sarah: > > > > This looks nice. It would be nice to have a plot showing Coaster > > activity, as well. (Jobs submitted, coasters allocated, etc.) Mihael, > > Mike: any chance of that? > > That would be useful. I'll see what I can do. Since swift r3164 there's a coasters tab in the log plots. Please test and report bugs. Note: this won't work on old logs. You need logs generated by swift >= r3164. Mihael From foster at anl.gov Thu Oct 15 23:36:32 2009 From: foster at anl.gov (Ian Foster) Date: Thu, 15 Oct 2009 23:36:32 -0500 Subject: [Swift-devel] burnin' up ranger w/the latest coasters In-Reply-To: <1255652500.9269.1.camel@localhost> References: <20091013111417.CDU59058@m4500-02.uchicago.edu> <075C230D-8AA8-467F-A07C-057A589EF762@anl.gov> <1255451231.12836.1.camel@localhost> <1255652500.9269.1.camel@localhost> Message-ID: <21142201-422A-4AF0-BE10-C91C97CC9180@anl.gov> Very nice! On Oct 15, 2009, at 7:21 PM, Mihael Hategan wrote: > On Tue, 2009-10-13 at 11:27 -0500, Mihael Hategan wrote: >> On Tue, 2009-10-13 at 11:23 -0500, Ian Foster wrote: >>> Sarah: >>> >>> This looks nice. It would be nice to have a plot showing Coaster >>> activity, as well. (Jobs submitted, coasters allocated, etc.) >>> Mihael, >>> Mike: any chance of that? >> >> That would be useful. I'll see what I can do. > > Since swift r3164 there's a coasters tab in the log plots. Please test > and report bugs. > > Note: this won't work on old logs. You need logs generated by swift >= > r3164. > > Mihael > From skenny at uchicago.edu Fri Oct 16 14:47:01 2009 From: skenny at uchicago.edu (skenny at uchicago.edu) Date: Fri, 16 Oct 2009 14:47:01 -0500 (CDT) Subject: [Swift-devel] burnin' up ranger w/the latest coasters In-Reply-To: <1255652500.9269.1.camel@localhost> References: <20091013111417.CDU59058@m4500-02.uchicago.edu> <075C230D-8AA8-467F-A07C-057A589EF762@anl.gov> <1255451231.12836.1.camel@localhost> <1255652500.9269.1.camel@localhost> Message-ID: <20091016144701.CDZ88537@m4500-02.uchicago.edu> >On Tue, 2009-10-13 at 11:27 -0500, Mihael Hategan wrote: >> On Tue, 2009-10-13 at 11:23 -0500, Ian Foster wrote: >> > Sarah: >> > >> > This looks nice. It would be nice to have a plot showing Coaster >> > activity, as well. (Jobs submitted, coasters allocated, etc.) Mihael, >> > Mike: any chance of that? >> >> That would be useful. I'll see what I can do. > >Since swift r3164 there's a coasters tab in the log plots. Please test >and report bugs. how can i make the logs less verbose? (getting >15G at this point) >Note: this won't work on old logs. You need logs generated by swift >= >r3164. > >Mihael > >_______________________________________________ >Swift-devel mailing list >Swift-devel at ci.uchicago.edu >http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel From hategan at mcs.anl.gov Fri Oct 16 14:51:00 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Fri, 16 Oct 2009 14:51:00 -0500 Subject: [Swift-devel] burnin' up ranger w/the latest coasters In-Reply-To: <20091016144701.CDZ88537@m4500-02.uchicago.edu> References: <20091013111417.CDU59058@m4500-02.uchicago.edu> <075C230D-8AA8-467F-A07C-057A589EF762@anl.gov> <1255451231.12836.1.camel@localhost> <1255652500.9269.1.camel@localhost> <20091016144701.CDZ88537@m4500-02.uchicago.edu> Message-ID: <1255722660.23045.0.camel@localhost> On Fri, 2009-10-16 at 14:47 -0500, skenny at uchicago.edu wrote: > >On Tue, 2009-10-13 at 11:27 -0500, Mihael Hategan wrote: > >> On Tue, 2009-10-13 at 11:23 -0500, Ian Foster wrote: > >> > Sarah: > >> > > >> > This looks nice. It would be nice to have a plot showing > Coaster > >> > activity, as well. (Jobs submitted, coasters allocated, > etc.) Mihael, > >> > Mike: any chance of that? > >> > >> That would be useful. I'll see what I can do. > > > >Since swift r3164 there's a coasters tab in the log plots. > Please test > >and report bugs. > > how can i make the logs less verbose? (getting >15G at this point) Oh, I promised you a solution there. Hang on. From hategan at mcs.anl.gov Fri Oct 16 15:16:08 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Fri, 16 Oct 2009 15:16:08 -0500 Subject: [Swift-devel] burnin' up ranger w/the latest coasters In-Reply-To: <20091016144701.CDZ88537@m4500-02.uchicago.edu> References: <20091013111417.CDU59058@m4500-02.uchicago.edu> <075C230D-8AA8-467F-A07C-057A589EF762@anl.gov> <1255451231.12836.1.camel@localhost> <1255652500.9269.1.camel@localhost> <20091016144701.CDZ88537@m4500-02.uchicago.edu> Message-ID: <1255724168.23366.1.camel@localhost> On Fri, 2009-10-16 at 14:47 -0500, skenny at uchicago.edu wrote: > how can i make the logs less verbose? (getting >15G at this point) > Swift r3166. Use "-reduced.logging" on the command line. What I need help with is test this to see if swift-plot-logs still works with that enabled. From hategan at mcs.anl.gov Fri Oct 16 17:02:39 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Fri, 16 Oct 2009 17:02:39 -0500 Subject: [Swift-devel] aliasing/copying Message-ID: <1255730559.12356.7.camel@localhost> In r3170 I added an implementation for (some) mapped-type assignments. This is an implementation for the following types of code: a = b; If a has a mapper supporting remapping (i.e. being told to map something to something after being initialized), then the effect is that of 'a' being a soft copy of 'b'. I implemented remapping support for the concurrent mapper. If a has a mapper that doesn't support remapping, b's file is copied to a's file. So: 1, file a <"test.txt">; file b; b = a; trace(@filename(b)) Will result in "SwiftScript trace: test.txt" 2. file a <"test.txt">; file b <"echo.txt">; b = a; trace(@filename(b)) Will result in "SwiftScript trace: echo.txt" and the contents of "test.txt" copied to "echo.txt". I have not run integration tests on this. It would be helpful if someone could. From aespinosa at cs.uchicago.edu Fri Oct 16 20:02:52 2009 From: aespinosa at cs.uchicago.edu (Allan Espinosa) Date: Fri, 16 Oct 2009 20:02:52 -0500 Subject: [Swift-devel] IO overheads of swift wrapper scripts on BlueGene/P Message-ID: <50b07b4b0910161802p70608e99k354b4b7cccc10b55@mail.gmail.com> The attached graph looks interesting. The green on the left shows the 192 job workload using the straight falkon client. The right one is through swift. Using pure falkon the total time for the workload is at around 814 seconds while the swift workflow took 1520 seconds. In some repeats of the same run, Swift took 3078 seconds. Just to confirm, in the deef-provider, the actual jobs that are dispatched to the Falkon workers include the ones in the _swiftwrap script correct? This must be the one inducing the overheads. From the top of my head create jobdir and link input dir can be the culprits. My workflow below does not do any input staging to the work directory. Only output staging is done which I believe is separate from the _swiftwrap script and will therefore not register time in the falkon graphs (or in EXECUTE state in swift). workflow: ype output; type std_err; type std_out; type BlastDatabase; type BlastQuery; type BlastResult { output out; std_err serr; std_out sout; } type BlastSummary; app (BlastResult out) blastall(string i, string db) { blastall "-p" "blastp" "-m" "8" "-e" "1.0e-5" "-FF" "-d" db "-i" i stdout=@filename(out.sout) "-o" @filename(out.out) stderr=@filename(out.serr); } string db="/intrepid-fs0/users/espinosa/persistent/datasets/nr_bob/nr_v200/nr"; BlastResult out[] ; BlastQuery input[] ; foreach data,i in input { out[i] = blastall(@strcat("/", at filename(data)), db); } The build of swift i'm using is fairly old (Swift svn swift-r2911 cog-r2394) -Allan -- Allan M. Espinosa PhD student, Computer Science University of Chicago -------------- next part -------------- A non-text attachment was scrubbed... Name: summary_graph.png Type: image/png Size: 5305 bytes Desc: not available URL: From hategan at mcs.anl.gov Fri Oct 16 20:31:37 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Fri, 16 Oct 2009 20:31:37 -0500 Subject: [Swift-devel] IO overheads of swift wrapper scripts on BlueGene/P In-Reply-To: <50b07b4b0910161802p70608e99k354b4b7cccc10b55@mail.gmail.com> References: <50b07b4b0910161802p70608e99k354b4b7cccc10b55@mail.gmail.com> Message-ID: <1255743097.16082.17.camel@localhost> On Fri, 2009-10-16 at 20:02 -0500, Allan Espinosa wrote: > The attached graph looks interesting. The green on the left shows the > 192 job workload using the straight falkon client. The right one is > through swift. Using pure falkon the total time for the workload is > at around 814 seconds while the swift workflow took 1520 seconds. In > some repeats of the same run, Swift took 3078 seconds. It would be useful to plot the swift stuff with swift-plot-log. It would also be useful to make use of the info files which record timing data for what the wrapper does. > > Just to confirm, in the deef-provider, the actual jobs that are > dispatched to the Falkon workers include the ones in the _swiftwrap > script correct? _swiftwrap is the job which forks blast (in your case). > This must be the one inducing the overheads. From > the top of my head create jobdir and link input dir can be the > culprits. Sounds plausible. To that add move-output-files-to-shared-dir. 3 per job as I see it. Again, info files should provide some details. > > My workflow below does not do any input staging to the work directory. > Only output staging is done which I believe is separate from the > _swiftwrap script and will therefore not register time in the falkon > graphs (or in EXECUTE state in swift). Right. Once _swiftwrap is done, the executor should be released, unless there's some other part in there that takes time (like, say, notification). Ioan would know this. From aespinosa at cs.uchicago.edu Fri Oct 16 21:07:16 2009 From: aespinosa at cs.uchicago.edu (Allan Espinosa) Date: Fri, 16 Oct 2009 21:07:16 -0500 Subject: [Swift-devel] IO overheads of swift wrapper scripts on BlueGene/P In-Reply-To: <1255743097.16082.17.camel@localhost> References: <50b07b4b0910161802p70608e99k354b4b7cccc10b55@mail.gmail.com> <1255743097.16082.17.camel@localhost> Message-ID: <50b07b4b0910161907u2f9a01bdpe2fafe2237d5b4e3@mail.gmail.com> The corresponding swift-plot-log is in http://www.ci.uchicago.edu/~aespinosa/swift/report-blastp-test3.4.7_3cpn.32ifs.192cpu/. I used the same runid for two runs so the statistics were merged. Just focus on the second lump of events that happened in the log. Here's the snippet of an info file. Progress 2009-10-16 17:30:44.576141000-0500 LOG_START _____________________________________________________________________________ Wrapper _____________________________________________________________________________ Job directory mode is: link on shared filesystem DIR=jobs/5/blastall-59j5q3ij EXEC=/home/espinosa/workflows/jgi_blastp/blastall_wrapper STDIN= STDOUT=home/espinosa/workflows/jgi_blastp/test3.4.7_3cpn.32ifs.192cpu/output/D00STDERR=home/espinosa/workflows/jgi_blastp/test3.4.7_3cpn.32ifs.192cpu/output/D00 DIRS=home/espinosa/workflows/jgi_blastp/test3.4.7_3cpn.32ifs.192cpu/output/D0000INF= OUTF=home/espinosa/workflows/jgi_blastp/test3.4.7_3cpn.32ifs.192cpu/output/D0000KICKSTART= ARGS=-p blastp -m 8 -e 1.0e-5 -FF -d /dataifs/nr -i /intrepid-fs0/users/espinosa ARGC=13 Progress 2009-10-16 17:30:46.378238000-0500 CREATE_JOBDIR Created job directory: jobs/5/blastall-59j5q3ij Progress 2009-10-16 17:30:59.498509000-0500 CREATE_INPUTDIR Created output directory: jobs/5/blastall-59j5q3ij/home/espinosa/workflows/jgi_b Progress 2009-10-16 17:33:25.777819000-0500 LINK_INPUTS Progress 2009-10-16 17:41:28.979051000-0500 EXECUTE Moving back to workflow directory /intrepid-fs0/users/espinosa/scratch/jgi-blastProgress 2009-10-16 17:58:11.732629000-0500 EXECUTE_DONE Job ran successfully Progress 2009-10-16 18:00:33.756364000-0500 COPYING_OUTPUTS Progress 2009-10-16 18:08:19.970449000-0500 RM_JOBDIR Progress 2009-10-16 18:08:57.155065000-0500 END from LOG_START to END, the time is around 2292.578924 seconds. the time between EXECUTE and EXECUTE_DONE is 1002.753578 which is fairly close to 891 seconds. It verifies that pre-execution states in _swiftwrap does add some overhead. specially in 256 cpus per PSET as were we are most likely starting to saturate the Tree network and the forwarding daemon. -Allan 2009/10/16 Mihael Hategan : > On Fri, 2009-10-16 at 20:02 -0500, Allan Espinosa wrote: >> The attached graph looks interesting. The green on the left shows the >> 192 job workload using the straight falkon client. The right one is >> through swift. ?Using pure falkon the total time for the workload is >> at around 814 seconds while the swift workflow took 1520 seconds. ?In >> some repeats of the same run, Swift took 3078 seconds. > > It would be useful to plot the swift stuff with swift-plot-log. > > It would also be useful to make use of the info files which record > timing data for what the wrapper does. > >> >> Just to confirm, in the deef-provider, the actual jobs that are >> dispatched to the Falkon workers include the ones in the _swiftwrap >> script correct? > > _swiftwrap is the job which forks blast (in your case). > >> ? This must be the one inducing the overheads. ?From >> the top of my head create jobdir and link input dir can be the >> culprits. > > Sounds plausible. > > To that add move-output-files-to-shared-dir. 3 per job as I see it. > Again, info files should provide some details. > >> >> My workflow below does not do any input staging to the work directory. >> ?Only output staging is done which I believe is separate from the >> _swiftwrap script and will therefore not register time in the falkon >> graphs (or in EXECUTE state in swift). > > Right. Once _swiftwrap is done, the executor should be released, unless > there's some other part in there that takes time (like, say, > notification). Ioan would know this. > > > -- Allan M. Espinosa PhD student, Computer Science University of Chicago From hategan at mcs.anl.gov Fri Oct 16 23:26:50 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Fri, 16 Oct 2009 23:26:50 -0500 Subject: [Swift-devel] IO overheads of swift wrapper scripts on BlueGene/P In-Reply-To: <50b07b4b0910161907u2f9a01bdpe2fafe2237d5b4e3@mail.gmail.com> References: <50b07b4b0910161802p70608e99k354b4b7cccc10b55@mail.gmail.com> <1255743097.16082.17.camel@localhost> <50b07b4b0910161907u2f9a01bdpe2fafe2237d5b4e3@mail.gmail.com> Message-ID: <1255753610.19114.9.camel@localhost> On Fri, 2009-10-16 at 21:07 -0500, Allan Espinosa wrote: > Progress 2009-10-16 18:00:33.756364000-0500 COPYING_OUTPUTS > Progress 2009-10-16 18:08:19.970449000-0500 RM_JOBDIR Grr. 8 minutes spent COPYING_OUTPUTS. What would be useful is to aggregate all the access that happened on that FS from all the relevant jobs, to see the exact thing that causes contention. I strongly suspect it's home/espinosa/workflows/jgi_blastp/test3.4.7_3cpn.32ifs.192cpu/output/ Pretty much all the outputs seem to go to that directory. I'm afraid however that the information in the logs is insufficient. Strace with relevant options (for fs calls only) may be useful if you want to try. Alternatively, you could try to spread your output over multiple directories and see what the difference is. Also, it may be interesting to see the dependence between the delay and the number of contending processes. That is so that we know the limit of how many processes we can allow to compete for a shared resource without causing too much trouble. Mihael From aespinosa at cs.uchicago.edu Sat Oct 17 02:59:10 2009 From: aespinosa at cs.uchicago.edu (Allan Espinosa) Date: Sat, 17 Oct 2009 02:59:10 -0500 Subject: [Swift-devel] IO overheads of swift wrapper scripts on BlueGene/P In-Reply-To: <1255753610.19114.9.camel@localhost> References: <50b07b4b0910161802p70608e99k354b4b7cccc10b55@mail.gmail.com> <1255743097.16082.17.camel@localhost> <50b07b4b0910161907u2f9a01bdpe2fafe2237d5b4e3@mail.gmail.com> <1255753610.19114.9.camel@localhost> Message-ID: <50b07b4b0910170059h65625594see9aeed8014179c9@mail.gmail.com> I was using 1000 files (or was it 3000?) per directory. it looks like i need to lower my ratio... -Allan 2009/10/16 Mihael Hategan : > On Fri, 2009-10-16 at 21:07 -0500, Allan Espinosa wrote: >> Progress ?2009-10-16 18:00:33.756364000-0500 ?COPYING_OUTPUTS >> Progress ?2009-10-16 18:08:19.970449000-0500 ?RM_JOBDIR > > Grr. 8 minutes spent COPYING_OUTPUTS. > > What would be useful is to aggregate all the access that happened on > that FS from all the relevant jobs, to see the exact thing that causes > contention. I strongly suspect it's > home/espinosa/workflows/jgi_blastp/test3.4.7_3cpn.32ifs.192cpu/output/ > > Pretty much all the outputs seem to go to that directory. > > I'm afraid however that the information in the logs is insufficient. > Strace with relevant options (for fs calls only) may be useful if you > want to try. > > Alternatively, you could try to spread your output over multiple > directories and see what the difference is. > > Also, it may be interesting to see the dependence between the delay and > the number of contending processes. That is so that we know the limit of > how many processes we can allow to compete for a shared resource without > causing too much trouble. > > Mihael > > > -- Allan M. Espinosa PhD student, Computer Science University of Chicago From wilde at mcs.anl.gov Sat Oct 17 12:29:38 2009 From: wilde at mcs.anl.gov (Michael Wilde) Date: Sat, 17 Oct 2009 12:29:38 -0500 Subject: [Swift-devel] IO overheads of swift wrapper scripts on BlueGene/P In-Reply-To: <50b07b4b0910170059h65625594see9aeed8014179c9@mail.gmail.com> References: <50b07b4b0910161802p70608e99k354b4b7cccc10b55@mail.gmail.com> <1255743097.16082.17.camel@localhost> <50b07b4b0910161907u2f9a01bdpe2fafe2237d5b4e3@mail.gmail.com> <1255753610.19114.9.camel@localhost> <50b07b4b0910170059h65625594see9aeed8014179c9@mail.gmail.com> Message-ID: <4AD9FF02.6060705@mcs.anl.gov> Remember that any situation in which multiple IONs modify the same file or directory (ie by creating files or directories in the same parent directory) will cause severe contention and performance degradation on any GPFS filesystem. In addition to creating many directories, you need to ensure that no single file or directories is likely to ever be written to from multiple client nodes (eg IONs on the BG/P) concurrently. Have you done that in this workload, Allan? - Mike On 10/17/09 2:59 AM, Allan Espinosa wrote: > I was using 1000 files (or was it 3000?) per directory. it looks like > i need to lower my ratio... > > -Allan > > 2009/10/16 Mihael Hategan : >> On Fri, 2009-10-16 at 21:07 -0500, Allan Espinosa wrote: >>> Progress 2009-10-16 18:00:33.756364000-0500 COPYING_OUTPUTS >>> Progress 2009-10-16 18:08:19.970449000-0500 RM_JOBDIR >> Grr. 8 minutes spent COPYING_OUTPUTS. >> >> What would be useful is to aggregate all the access that happened on >> that FS from all the relevant jobs, to see the exact thing that causes >> contention. I strongly suspect it's >> home/espinosa/workflows/jgi_blastp/test3.4.7_3cpn.32ifs.192cpu/output/ >> >> Pretty much all the outputs seem to go to that directory. >> >> I'm afraid however that the information in the logs is insufficient. >> Strace with relevant options (for fs calls only) may be useful if you >> want to try. >> >> Alternatively, you could try to spread your output over multiple >> directories and see what the difference is. >> >> Also, it may be interesting to see the dependence between the delay and >> the number of contending processes. That is so that we know the limit of >> how many processes we can allow to compete for a shared resource without >> causing too much trouble. >> >> Mihael >> >> >> > > > From aespinosa at cs.uchicago.edu Sat Oct 17 21:57:45 2009 From: aespinosa at cs.uchicago.edu (Allan Espinosa) Date: Sat, 17 Oct 2009 21:57:45 -0500 Subject: [Swift-devel] IO overheads of swift wrapper scripts on BlueGene/P In-Reply-To: <4AD9FF02.6060705@mcs.anl.gov> References: <50b07b4b0910161802p70608e99k354b4b7cccc10b55@mail.gmail.com> <1255743097.16082.17.camel@localhost> <50b07b4b0910161907u2f9a01bdpe2fafe2237d5b4e3@mail.gmail.com> <1255753610.19114.9.camel@localhost> <50b07b4b0910170059h65625594see9aeed8014179c9@mail.gmail.com> <4AD9FF02.6060705@mcs.anl.gov> Message-ID: <50b07b4b0910171957of897b9dy90c349d1069004be@mail.gmail.com> Here I tried one directory per job (Q0000130). 3 output files are expected per directory which are produced by a single job: Progress 2009-10-17 20:53:56.943503000-0500 LOG_START _____________________________________________________________________________ Wrapper _____________________________________________________________________________ Job directory mode is: link on shared filesystem DIR=jobs/7/blastall-715ul5ij EXEC=/home/espinosa/workflows/jgi_blastp/blastall_wrapper STDIN= STDOUT=home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130/out_Q0000130.sout STDERR=home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130/out_Q0000130.serr DIRS=home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130 INF= OUTF=home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130/out_Q0000130.out^home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130/out_Q0000130.serr^home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130/out_Q0000130.sout KICKSTART= ARGS=-p blastp -m 8 -e 1.0e-5 -FF -d /dataifs/nr -i /intrepid-fs0/users/espinosa/persistent/datasets/nr_bob/queries/mock_2seq/D0000000/SEQ0000130.fasta -o home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130/out_Q0000130.out ARGC=13 Progress 2009-10-17 20:53:58.656335000-0500 CREATE_JOBDIR Created job directory: jobs/7/blastall-715ul5ij Progress 2009-10-17 20:54:05.204962000-0500 CREATE_INPUTDIR Created output directory: jobs/7/blastall-715ul5ij/home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130 Progress 2009-10-17 20:54:15.498666000-0500 LINK_INPUTS Progress 2009-10-17 20:54:19.900786000-0500 EXECUTE Moving back to workflow directory /fuse/intrepid-fs0/users/espinosa/scratch/jgi-blastp_runs/blastp-test3.2.7_3cpn.64ifs.192cpu Progress 2009-10-17 21:20:23.390800000-0500 EXECUTE_DONE Job ran successfully Progress 2009-10-17 21:31:11.179664000-0500 COPYING_OUTPUTS Progress 2009-10-17 21:37:14.539569000-0500 RM_JOBDIR Progress 2009-10-17 21:38:24.220130000-0500 END COPYING_OUTPUTS still take time. 2009/10/17 Michael Wilde : > Remember that any situation in which multiple IONs modify the same file or > directory (ie by creating files or directories in the same parent directory) > will cause severe contention and performance degradation on any GPFS > filesystem. > > In addition to creating many directories, you need to ensure that no single > file or directories is likely to ever be written to from multiple client > nodes (eg IONs on the BG/P) concurrently. This workload is just over 1 PSET so there are no other IONs contending over the directories. > > Have you done that in this workload, Allan? > > - Mike > > > On 10/17/09 2:59 AM, Allan Espinosa wrote: >> >> I was using 1000 files ?(or was it 3000?) per directory. it looks like >> i need to lower my ratio... >> >> -Allan >> >> 2009/10/16 Mihael Hategan : >>> >>> On Fri, 2009-10-16 at 21:07 -0500, Allan Espinosa wrote: >>>> >>>> Progress ?2009-10-16 18:00:33.756364000-0500 ?COPYING_OUTPUTS >>>> Progress ?2009-10-16 18:08:19.970449000-0500 ?RM_JOBDIR >>> >>> Grr. 8 minutes spent COPYING_OUTPUTS. >>> >>> What would be useful is to aggregate all the access that happened on >>> that FS from all the relevant jobs, to see the exact thing that causes >>> contention. I strongly suspect it's >>> home/espinosa/workflows/jgi_blastp/test3.4.7_3cpn.32ifs.192cpu/output/ >>> >>> Pretty much all the outputs seem to go to that directory. >>> >>> I'm afraid however that the information in the logs is insufficient. >>> Strace with relevant options (for fs calls only) may be useful if you >>> want to try. >>> >>> Alternatively, you could try to spread your output over multiple >>> directories and see what the difference is. >>> >>> Also, it may be interesting to see the dependence between the delay and >>> the number of contending processes. That is so that we know the limit of >>> how many processes we can allow to compete for a shared resource without >>> causing too much trouble. >>> >>> Mihael >>> >>> >>> >> >> >> > > -- Allan M. Espinosa PhD student, Computer Science University of Chicago From iraicu at cs.uchicago.edu Sun Oct 18 09:06:47 2009 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Sun, 18 Oct 2009 09:06:47 -0500 Subject: [Swift-devel] IO overheads of swift wrapper scripts on BlueGene/P In-Reply-To: <50b07b4b0910171957of897b9dy90c349d1069004be@mail.gmail.com> References: <50b07b4b0910161802p70608e99k354b4b7cccc10b55@mail.gmail.com> <1255743097.16082.17.camel@localhost> <50b07b4b0910161907u2f9a01bdpe2fafe2237d5b4e3@mail.gmail.com> <1255753610.19114.9.camel@localhost> <50b07b4b0910170059h65625594see9aeed8014179c9@mail.gmail.com> <4AD9FF02.6060705@mcs.anl.gov> <50b07b4b0910171957of897b9dy90c349d1069004be@mail.gmail.com> Message-ID: <4ADB20F7.80000@cs.uchicago.edu> Hi Allan, I don't remember, but your Falkon only run seemed to run OK, right? Didn't that also produce the output files Swift is producing? Or is Swift doing an extra step, to copy/move files from one place to another after the computation terminates, which is the thing that takes so long? Just trying to understand the difference between the Falkon only run and Swift run. Ioan -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= Allan Espinosa wrote: > Here I tried one directory per job (Q0000130). 3 output files are > expected per directory which are produced by a single job: > > Progress 2009-10-17 20:53:56.943503000-0500 LOG_START > > _____________________________________________________________________________ > > Wrapper > _____________________________________________________________________________ > > Job directory mode is: link on shared filesystem > DIR=jobs/7/blastall-715ul5ij > EXEC=/home/espinosa/workflows/jgi_blastp/blastall_wrapper > STDIN= > STDOUT=home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130/out_Q0000130.sout > STDERR=home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130/out_Q0000130.serr > DIRS=home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130 > INF= > OUTF=home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130/out_Q0000130.out^home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130/out_Q0000130.serr^home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130/out_Q0000130.sout > KICKSTART= > ARGS=-p blastp -m 8 -e 1.0e-5 -FF -d /dataifs/nr -i > /intrepid-fs0/users/espinosa/persistent/datasets/nr_bob/queries/mock_2seq/D0000000/SEQ0000130.fasta > -o home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130/out_Q0000130.out > ARGC=13 > Progress 2009-10-17 20:53:58.656335000-0500 CREATE_JOBDIR > Created job directory: jobs/7/blastall-715ul5ij > Progress 2009-10-17 20:54:05.204962000-0500 CREATE_INPUTDIR > Created output directory: > jobs/7/blastall-715ul5ij/home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130 > Progress 2009-10-17 20:54:15.498666000-0500 LINK_INPUTS > Progress 2009-10-17 20:54:19.900786000-0500 EXECUTE > Moving back to workflow directory > /fuse/intrepid-fs0/users/espinosa/scratch/jgi-blastp_runs/blastp-test3.2.7_3cpn.64ifs.192cpu > Progress 2009-10-17 21:20:23.390800000-0500 EXECUTE_DONE > Job ran successfully > Progress 2009-10-17 21:31:11.179664000-0500 COPYING_OUTPUTS > Progress 2009-10-17 21:37:14.539569000-0500 RM_JOBDIR > Progress 2009-10-17 21:38:24.220130000-0500 END > > > COPYING_OUTPUTS still take time. > > 2009/10/17 Michael Wilde : > >> Remember that any situation in which multiple IONs modify the same file or >> directory (ie by creating files or directories in the same parent directory) >> will cause severe contention and performance degradation on any GPFS >> filesystem. >> >> In addition to creating many directories, you need to ensure that no single >> file or directories is likely to ever be written to from multiple client >> nodes (eg IONs on the BG/P) concurrently. >> > > This workload is just over 1 PSET so there are no other IONs > contending over the directories. > > >> Have you done that in this workload, Allan? >> >> - Mike >> >> >> On 10/17/09 2:59 AM, Allan Espinosa wrote: >> >>> I was using 1000 files (or was it 3000?) per directory. it looks like >>> i need to lower my ratio... >>> >>> -Allan >>> >>> 2009/10/16 Mihael Hategan : >>> >>>> On Fri, 2009-10-16 at 21:07 -0500, Allan Espinosa wrote: >>>> >>>>> Progress 2009-10-16 18:00:33.756364000-0500 COPYING_OUTPUTS >>>>> Progress 2009-10-16 18:08:19.970449000-0500 RM_JOBDIR >>>>> >>>> Grr. 8 minutes spent COPYING_OUTPUTS. >>>> >>>> What would be useful is to aggregate all the access that happened on >>>> that FS from all the relevant jobs, to see the exact thing that causes >>>> contention. I strongly suspect it's >>>> home/espinosa/workflows/jgi_blastp/test3.4.7_3cpn.32ifs.192cpu/output/ >>>> >>>> Pretty much all the outputs seem to go to that directory. >>>> >>>> I'm afraid however that the information in the logs is insufficient. >>>> Strace with relevant options (for fs calls only) may be useful if you >>>> want to try. >>>> >>>> Alternatively, you could try to spread your output over multiple >>>> directories and see what the difference is. >>>> >>>> Also, it may be interesting to see the dependence between the delay and >>>> the number of contending processes. That is so that we know the limit of >>>> how many processes we can allow to compete for a shared resource without >>>> causing too much trouble. >>>> >>>> Mihael >>>> >>>> >>>> >>>> >>> >>> >> > > > > -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From aespinosa at cs.uchicago.edu Sun Oct 18 09:24:35 2009 From: aespinosa at cs.uchicago.edu (Allan Espinosa) Date: Sun, 18 Oct 2009 09:24:35 -0500 Subject: [Swift-devel] IO overheads of swift wrapper scripts on BlueGene/P In-Reply-To: <4ADB20F7.80000@cs.uchicago.edu> References: <50b07b4b0910161802p70608e99k354b4b7cccc10b55@mail.gmail.com> <1255743097.16082.17.camel@localhost> <50b07b4b0910161907u2f9a01bdpe2fafe2237d5b4e3@mail.gmail.com> <1255753610.19114.9.camel@localhost> <50b07b4b0910170059h65625594see9aeed8014179c9@mail.gmail.com> <4AD9FF02.6060705@mcs.anl.gov> <50b07b4b0910171957of897b9dy90c349d1069004be@mail.gmail.com> <4ADB20F7.80000@cs.uchicago.edu> Message-ID: <50b07b4b0910180724q630bb22dve489bcad105bd824@mail.gmail.com> swift does extra moves from job directory to work directory. which takes long in this case. -Allan 2009/10/18 Ioan Raicu : > Hi Allan, > I don't remember, but your Falkon only run seemed to run OK, right? Didn't > that also produce the output files Swift is producing? Or is Swift doing an > extra step, to copy/move files from one place to another after the > computation terminates, which is the thing that takes so long? Just trying > to understand the difference between the Falkon only run and Swift run. > > Ioan > > -- > ================================================================= > Ioan Raicu, Ph.D. > NSF/CRA Computing Innovation Fellow > ================================================================= > Center for Ultra-scale Computing and Information Security (CUCIS) > Department of Electrical Engineering and Computer Science > Northwestern University > 2145 Sheridan Rd, Tech M384 > Evanston, IL 60208-3118 > ================================================================= > Cel: 1-847-722-0876 > Tel: 1-847-491-8163 > Email: iraicu at eecs.northwestern.edu > Web: http://www.eecs.northwestern.edu/~iraicu/ > https://wiki.cucis.eecs.northwestern.edu/ > ================================================================= > ================================================================= > > > > Allan Espinosa wrote: > > Here I tried one directory per job (Q0000130). 3 output files are > expected per directory which are produced by a single job: > > Progress 2009-10-17 20:53:56.943503000-0500 LOG_START > > _____________________________________________________________________________ > > Wrapper > _____________________________________________________________________________ > > Job directory mode is: link on shared filesystem > DIR=jobs/7/blastall-715ul5ij > EXEC=/home/espinosa/workflows/jgi_blastp/blastall_wrapper > STDIN= > STDOUT=home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130/out_Q0000130.sout > STDERR=home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130/out_Q0000130.serr > DIRS=home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130 > INF= > OUTF=home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130/out_Q0000130.out^home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130/out_Q0000130.serr^home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130/out_Q0000130.sout > KICKSTART= > ARGS=-p blastp -m 8 -e 1.0e-5 -FF -d /dataifs/nr -i > /intrepid-fs0/users/espinosa/persistent/datasets/nr_bob/queries/mock_2seq/D0000000/SEQ0000130.fasta > -o > home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130/out_Q0000130.out > ARGC=13 > Progress 2009-10-17 20:53:58.656335000-0500 CREATE_JOBDIR > Created job directory: jobs/7/blastall-715ul5ij > Progress 2009-10-17 20:54:05.204962000-0500 CREATE_INPUTDIR > Created output directory: > jobs/7/blastall-715ul5ij/home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130 > Progress 2009-10-17 20:54:15.498666000-0500 LINK_INPUTS > Progress 2009-10-17 20:54:19.900786000-0500 EXECUTE > Moving back to workflow directory > /fuse/intrepid-fs0/users/espinosa/scratch/jgi-blastp_runs/blastp-test3.2.7_3cpn.64ifs.192cpu > Progress 2009-10-17 21:20:23.390800000-0500 EXECUTE_DONE > Job ran successfully > Progress 2009-10-17 21:31:11.179664000-0500 COPYING_OUTPUTS > Progress 2009-10-17 21:37:14.539569000-0500 RM_JOBDIR > Progress 2009-10-17 21:38:24.220130000-0500 END > > > COPYING_OUTPUTS still take time. > > 2009/10/17 Michael Wilde : > > > Remember that any situation in which multiple IONs modify the same file or > directory (ie by creating files or directories in the same parent directory) > will cause severe contention and performance degradation on any GPFS > filesystem. > > In addition to creating many directories, you need to ensure that no single > file or directories is likely to ever be written to from multiple client > nodes (eg IONs on the BG/P) concurrently. > > > This workload is just over 1 PSET so there are no other IONs > contending over the directories. > > > > Have you done that in this workload, Allan? > > - Mike > > > On 10/17/09 2:59 AM, Allan Espinosa wrote: > > > I was using 1000 files ?(or was it 3000?) per directory. it looks like > i need to lower my ratio... > > -Allan > > 2009/10/16 Mihael Hategan : > > > On Fri, 2009-10-16 at 21:07 -0500, Allan Espinosa wrote: > > > Progress ?2009-10-16 18:00:33.756364000-0500 ?COPYING_OUTPUTS > Progress ?2009-10-16 18:08:19.970449000-0500 ?RM_JOBDIR > > > Grr. 8 minutes spent COPYING_OUTPUTS. > > What would be useful is to aggregate all the access that happened on > that FS from all the relevant jobs, to see the exact thing that causes > contention. I strongly suspect it's > home/espinosa/workflows/jgi_blastp/test3.4.7_3cpn.32ifs.192cpu/output/ > > Pretty much all the outputs seem to go to that directory. > > I'm afraid however that the information in the logs is insufficient. > Strace with relevant options (for fs calls only) may be useful if you > want to try. > > Alternatively, you could try to spread your output over multiple > directories and see what the difference is. > > Also, it may be interesting to see the dependence between the delay and > the number of contending processes. That is so that we know the limit of > how many processes we can allow to compete for a shared resource without > causing too much trouble. > > Mihael > > > > > > > > > =============================== From hategan at mcs.anl.gov Sun Oct 18 12:08:48 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sun, 18 Oct 2009 12:08:48 -0500 Subject: [Swift-devel] IO overheads of swift wrapper scripts on BlueGene/P In-Reply-To: <50b07b4b0910171957of897b9dy90c349d1069004be@mail.gmail.com> References: <50b07b4b0910161802p70608e99k354b4b7cccc10b55@mail.gmail.com> <1255743097.16082.17.camel@localhost> <50b07b4b0910161907u2f9a01bdpe2fafe2237d5b4e3@mail.gmail.com> <1255753610.19114.9.camel@localhost> <50b07b4b0910170059h65625594see9aeed8014179c9@mail.gmail.com> <4AD9FF02.6060705@mcs.anl.gov> <50b07b4b0910171957of897b9dy90c349d1069004be@mail.gmail.com> Message-ID: <1255885729.30428.6.camel@localhost> On Sat, 2009-10-17 at 21:57 -0500, Allan Espinosa wrote: > Progress 2009-10-17 20:53:58.656335000-0500 CREATE_JOBDIR > Created job directory: jobs/7/blastall-715ul5ij > Progress 2009-10-17 20:54:05.204962000-0500 CREATE_INPUTDIR > Created output directory: > jobs/7/blastall-715ul5ij/home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130 > Progress 2009-10-17 20:54:15.498666000-0500 LINK_INPUTS > Progress 2009-10-17 20:54:19.900786000-0500 EXECUTE > Moving back to workflow directory > /fuse/intrepid-fs0/users/espinosa/scratch/jgi-blastp_runs/blastp-test3.2.7_3cpn.64ifs.192cpu > Progress 2009-10-17 21:20:23.390800000-0500 EXECUTE_DONE > Job ran successfully > Progress 2009-10-17 21:31:11.179664000-0500 COPYING_OUTPUTS > Progress 2009-10-17 21:37:14.539569000-0500 RM_JOBDIR > Progress 2009-10-17 21:38:24.220130000-0500 END > > > COPYING_OUTPUTS still take time. Isn't that the curiousest thing. Something smells broken on that machine. From hategan at mcs.anl.gov Sun Oct 18 12:38:09 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sun, 18 Oct 2009 12:38:09 -0500 Subject: [Swift-devel] IO overheads of swift wrapper scripts on BlueGene/P In-Reply-To: <50b07b4b0910171957of897b9dy90c349d1069004be@mail.gmail.com> References: <50b07b4b0910161802p70608e99k354b4b7cccc10b55@mail.gmail.com> <1255743097.16082.17.camel@localhost> <50b07b4b0910161907u2f9a01bdpe2fafe2237d5b4e3@mail.gmail.com> <1255753610.19114.9.camel@localhost> <50b07b4b0910170059h65625594see9aeed8014179c9@mail.gmail.com> <4AD9FF02.6060705@mcs.anl.gov> <50b07b4b0910171957of897b9dy90c349d1069004be@mail.gmail.com> Message-ID: <1255887489.476.5.camel@localhost> On Sat, 2009-10-17 at 21:57 -0500, Allan Espinosa wrote: > Progress 2009-10-17 21:31:11.179664000-0500 COPYING_OUTPUTS > Progress 2009-10-17 21:37:14.539569000-0500 RM_JOBDIR Can you modify the wrapper to do an strace on the mv and the swift script to also stage out the output of that? I would expect that if the writing to gpfs is the problem, then writing to the info file would also take a long time, which doesn't seem to be the case: Progress 2009-10-17 20:54:15.498666000-0500 LINK_INPUTS Progress 2009-10-17 20:54:19.900786000-0500 EXECUTE Mihael From hategan at mcs.anl.gov Mon Oct 19 00:32:10 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 19 Oct 2009 00:32:10 -0500 Subject: [Swift-devel] [Fwd: re: IO overheads of swift wrapper scripts on BlueGene/P (fwd)] Message-ID: <1255930330.7690.0.camel@localhost> This from Ben. -------- Forwarded Message -------- From: Ben Clifford To: hategan at mcs.anl.gov Subject: re: IO overheads of swift wrapper scripts on BlueGene/P (fwd) Date: Mon, 19 Oct 2009 03:28:07 +0000 (GMT) i sent this earlier then i remembered that no one seems to moderate-approve my swift devel postings ever... ---------- Forwarded message ---------- Date: Mon, 19 Oct 2009 01:59:44 +0000 (GMT) From: Ben Clifford To: swift-devel at ci.uchicago.edu Subject: re: IO overheads of swift wrapper scripts on BlueGene/P swift r2970 changed copying outputs to a hard-link of outputs to improve performance at the output stage. you might be interested to see how that works. (also, don't use the same runid for different runs. you'll get into trouble somehow sooner or later. ) r2970 | benc | 2009-06-22 23:40:53 +1000 (Mon, 22 Jun 2009) | 6 lines Make wrapper use mv to stage out files to shared directory rather than cp. This should be faster when the job directories are on the same filesystem as the site shared cache. (suggested by tuecke) From hategan at mcs.anl.gov Mon Oct 19 00:39:16 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 19 Oct 2009 00:39:16 -0500 Subject: [Swift-devel] [Fwd: re: IO overheads of swift wrapper scripts on BlueGene/P (fwd)] In-Reply-To: <1255930330.7690.0.camel@localhost> References: <1255930330.7690.0.camel@localhost> Message-ID: <1255930756.7690.7.camel@localhost> On Mon, 2009-10-19 at 00:32 -0500, Mihael Hategan wrote: > -------- Forwarded Message -------- > From: Ben Clifford > To: hategan at mcs.anl.gov > Subject: re: IO overheads of swift wrapper scripts on BlueGene/P (fwd) > Date: Mon, 19 Oct 2009 03:28:07 +0000 (GMT) > > i sent this earlier then i remembered that no one seems to > moderate-approve my swift devel postings ever... I don't seem to be getting those. On this topic, why did we unsubscribe one of swift's committers from the swift mailing list(s)? > > ---------- Forwarded message ---------- > Date: Mon, 19 Oct 2009 01:59:44 +0000 (GMT) > From: Ben Clifford > To: swift-devel at ci.uchicago.edu > Subject: re: IO overheads of swift wrapper scripts on BlueGene/P > > > swift r2970 changed copying outputs to a hard-link of outputs to improve > performance at the output stage. you might be interested to see how that > works. > > (also, don't use the same runid for different runs. you'll get into > trouble somehow sooner or later. ) > > > r2970 | benc | 2009-06-22 23:40:53 +1000 (Mon, 22 Jun 2009) | 6 lines > > Make wrapper use mv to stage out files to shared directory rather than cp. > This should be faster when the job directories are on the same filesystem > as the site shared cache. > > (suggested by tuecke) Right. We're already using that. mv, on the same fs, would just make an entry in the new directory pointing to the same data instead of duplicating the data. Though I suppose this depends on the FS. However, we have to filesystems here, so it doesn't apply. From hategan at mcs.anl.gov Mon Oct 19 01:24:24 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 19 Oct 2009 01:24:24 -0500 Subject: [Swift-devel] [Fwd: re: IO overheads of swift wrapper scripts on BlueGene/P (fwd)] In-Reply-To: References: <1255930330.7690.0.camel@localhost> <1255930756.7690.7.camel@localhost> Message-ID: <1255933464.8032.26.camel@localhost> On Mon, 2009-10-19 at 05:49 +0000, Ben Clifford wrote: > I guess any posixy file system would work ok. Makes me wonder how ln (and > hard-links in general) behave(s) on an MS-DOS filesystem mounted under > linux "Operation not supported"? From wilde at mcs.anl.gov Mon Oct 19 07:59:12 2009 From: wilde at mcs.anl.gov (Michael Wilde) Date: Mon, 19 Oct 2009 07:59:12 -0500 Subject: [Swift-devel] [Fwd: re: IO overheads of swift wrapper scripts on BlueGene/P (fwd)] In-Reply-To: <1255930756.7690.7.camel@localhost> References: <1255930330.7690.0.camel@localhost> <1255930756.7690.7.camel@localhost> Message-ID: <4ADC62A0.8030407@mcs.anl.gov> On 10/19/09 12:39 AM, Mihael Hategan wrote: > On Mon, 2009-10-19 at 00:32 -0500, Mihael Hategan wrote: >> -------- Forwarded Message -------- >> From: Ben Clifford >> To: hategan at mcs.anl.gov >> Subject: re: IO overheads of swift wrapper scripts on BlueGene/P (fwd) >> Date: Mon, 19 Oct 2009 03:28:07 +0000 (GMT) >> >> i sent this earlier then i remembered that no one seems to >> moderate-approve my swift devel postings ever... I do, when I see them. > > I don't seem to be getting those. > > On this topic, why did we unsubscribe one of swift's committers from the > swift mailing list(s)? I thought this was done by Ben. He's certainly welcome to subscribe. > >> ---------- Forwarded message ---------- >> Date: Mon, 19 Oct 2009 01:59:44 +0000 (GMT) >> From: Ben Clifford >> To: swift-devel at ci.uchicago.edu >> Subject: re: IO overheads of swift wrapper scripts on BlueGene/P >> >> >> swift r2970 changed copying outputs to a hard-link of outputs to improve >> performance at the output stage. you might be interested to see how that >> works. >> >> (also, don't use the same runid for different runs. you'll get into >> trouble somehow sooner or later. ) >> >> >> r2970 | benc | 2009-06-22 23:40:53 +1000 (Mon, 22 Jun 2009) | 6 lines >> >> Make wrapper use mv to stage out files to shared directory rather than cp. >> This should be faster when the job directories are on the same filesystem >> as the site shared cache. >> >> (suggested by tuecke) > > > Right. We're already using that. mv, on the same fs, would just make an > entry in the new directory pointing to the same data instead of > duplicating the data. Though I suppose this depends on the FS. > > However, we have to filesystems here, so it doesn't apply. > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel From iraicu at cs.uchicago.edu Mon Oct 19 10:25:15 2009 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Mon, 19 Oct 2009 10:25:15 -0500 Subject: [Swift-devel] IO overheads of swift wrapper scripts on BlueGene/P In-Reply-To: <50b07b4b0910180724q630bb22dve489bcad105bd824@mail.gmail.com> References: <50b07b4b0910161802p70608e99k354b4b7cccc10b55@mail.gmail.com> <1255743097.16082.17.camel@localhost> <50b07b4b0910161907u2f9a01bdpe2fafe2237d5b4e3@mail.gmail.com> <1255753610.19114.9.camel@localhost> <50b07b4b0910170059h65625594see9aeed8014179c9@mail.gmail.com> <4AD9FF02.6060705@mcs.anl.gov> <50b07b4b0910171957of897b9dy90c349d1069004be@mail.gmail.com> <4ADB20F7.80000@cs.uchicago.edu> <50b07b4b0910180724q630bb22dve489bcad105bd824@mail.gmail.com> Message-ID: <4ADC84DB.6080608@cs.uchicago.edu> OK, I see now. The move in theory should be light weight, right? As its just metadata that is changing (e.g. moving on the same filesystem, not copying), right? Or is the job dir really in the compute node RAM, and the move is actually doing a copy from CN RAM to GPFS? Ioan Allan Espinosa wrote: > swift does extra moves from job directory to work directory. which > takes long in this case. > > -Allan > > 2009/10/18 Ioan Raicu : > >> Hi Allan, >> I don't remember, but your Falkon only run seemed to run OK, right? Didn't >> that also produce the output files Swift is producing? Or is Swift doing an >> extra step, to copy/move files from one place to another after the >> computation terminates, which is the thing that takes so long? Just trying >> to understand the difference between the Falkon only run and Swift run. >> >> Ioan >> >> -- >> ================================================================= >> Ioan Raicu, Ph.D. >> NSF/CRA Computing Innovation Fellow >> ================================================================= >> Center for Ultra-scale Computing and Information Security (CUCIS) >> Department of Electrical Engineering and Computer Science >> Northwestern University >> 2145 Sheridan Rd, Tech M384 >> Evanston, IL 60208-3118 >> ================================================================= >> Cel: 1-847-722-0876 >> Tel: 1-847-491-8163 >> Email: iraicu at eecs.northwestern.edu >> Web: http://www.eecs.northwestern.edu/~iraicu/ >> https://wiki.cucis.eecs.northwestern.edu/ >> ================================================================= >> ================================================================= >> >> >> >> Allan Espinosa wrote: >> >> Here I tried one directory per job (Q0000130). 3 output files are >> expected per directory which are produced by a single job: >> >> Progress 2009-10-17 20:53:56.943503000-0500 LOG_START >> >> _____________________________________________________________________________ >> >> Wrapper >> _____________________________________________________________________________ >> >> Job directory mode is: link on shared filesystem >> DIR=jobs/7/blastall-715ul5ij >> EXEC=/home/espinosa/workflows/jgi_blastp/blastall_wrapper >> STDIN= >> STDOUT=home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130/out_Q0000130.sout >> STDERR=home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130/out_Q0000130.serr >> DIRS=home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130 >> INF= >> OUTF=home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130/out_Q0000130.out^home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130/out_Q0000130.serr^home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130/out_Q0000130.sout >> KICKSTART= >> ARGS=-p blastp -m 8 -e 1.0e-5 -FF -d /dataifs/nr -i >> /intrepid-fs0/users/espinosa/persistent/datasets/nr_bob/queries/mock_2seq/D0000000/SEQ0000130.fasta >> -o >> home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130/out_Q0000130.out >> ARGC=13 >> Progress 2009-10-17 20:53:58.656335000-0500 CREATE_JOBDIR >> Created job directory: jobs/7/blastall-715ul5ij >> Progress 2009-10-17 20:54:05.204962000-0500 CREATE_INPUTDIR >> Created output directory: >> jobs/7/blastall-715ul5ij/home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130 >> Progress 2009-10-17 20:54:15.498666000-0500 LINK_INPUTS >> Progress 2009-10-17 20:54:19.900786000-0500 EXECUTE >> Moving back to workflow directory >> /fuse/intrepid-fs0/users/espinosa/scratch/jgi-blastp_runs/blastp-test3.2.7_3cpn.64ifs.192cpu >> Progress 2009-10-17 21:20:23.390800000-0500 EXECUTE_DONE >> Job ran successfully >> Progress 2009-10-17 21:31:11.179664000-0500 COPYING_OUTPUTS >> Progress 2009-10-17 21:37:14.539569000-0500 RM_JOBDIR >> Progress 2009-10-17 21:38:24.220130000-0500 END >> >> >> COPYING_OUTPUTS still take time. >> >> 2009/10/17 Michael Wilde : >> >> >> Remember that any situation in which multiple IONs modify the same file or >> directory (ie by creating files or directories in the same parent directory) >> will cause severe contention and performance degradation on any GPFS >> filesystem. >> >> In addition to creating many directories, you need to ensure that no single >> file or directories is likely to ever be written to from multiple client >> nodes (eg IONs on the BG/P) concurrently. >> >> >> This workload is just over 1 PSET so there are no other IONs >> contending over the directories. >> >> >> >> Have you done that in this workload, Allan? >> >> - Mike >> >> >> On 10/17/09 2:59 AM, Allan Espinosa wrote: >> >> >> I was using 1000 files (or was it 3000?) per directory. it looks like >> i need to lower my ratio... >> >> -Allan >> >> 2009/10/16 Mihael Hategan : >> >> >> On Fri, 2009-10-16 at 21:07 -0500, Allan Espinosa wrote: >> >> >> Progress 2009-10-16 18:00:33.756364000-0500 COPYING_OUTPUTS >> Progress 2009-10-16 18:08:19.970449000-0500 RM_JOBDIR >> >> >> Grr. 8 minutes spent COPYING_OUTPUTS. >> >> What would be useful is to aggregate all the access that happened on >> that FS from all the relevant jobs, to see the exact thing that causes >> contention. I strongly suspect it's >> home/espinosa/workflows/jgi_blastp/test3.4.7_3cpn.32ifs.192cpu/output/ >> >> Pretty much all the outputs seem to go to that directory. >> >> I'm afraid however that the information in the logs is insufficient. >> Strace with relevant options (for fs calls only) may be useful if you >> want to try. >> >> Alternatively, you could try to spread your output over multiple >> directories and see what the difference is. >> >> Also, it may be interesting to see the dependence between the delay and >> the number of contending processes. That is so that we know the limit of >> how many processes we can allow to compete for a shared resource without >> causing too much trouble. >> >> Mihael >> >> >> >> >> >> >> >> >> >> > > =============================== > > -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Mon Oct 19 10:34:41 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 19 Oct 2009 10:34:41 -0500 Subject: [Swift-devel] IO overheads of swift wrapper scripts on BlueGene/P In-Reply-To: <4ADC84DB.6080608@cs.uchicago.edu> References: <50b07b4b0910161802p70608e99k354b4b7cccc10b55@mail.gmail.com> <1255743097.16082.17.camel@localhost> <50b07b4b0910161907u2f9a01bdpe2fafe2237d5b4e3@mail.gmail.com> <1255753610.19114.9.camel@localhost> <50b07b4b0910170059h65625594see9aeed8014179c9@mail.gmail.com> <4AD9FF02.6060705@mcs.anl.gov> <50b07b4b0910171957of897b9dy90c349d1069004be@mail.gmail.com> <4ADB20F7.80000@cs.uchicago.edu> <50b07b4b0910180724q630bb22dve489bcad105bd824@mail.gmail.com> <4ADC84DB.6080608@cs.uchicago.edu> Message-ID: <1255966481.9399.1.camel@localhost> On Mon, 2009-10-19 at 10:25 -0500, Ioan Raicu wrote: > OK, I see now. The move in theory should be light weight, right? As > its just metadata that is changing (e.g. moving on the same > filesystem, not copying), right? Or is the job dir really in the > compute node RAM, and the move is actually doing a copy from CN RAM to > GPFS? In Allan's case, I believe it's scratch -> gpfs. It's still troubling to see 3 small files taking 3 minutes to be copied. From wilde at mcs.anl.gov Mon Oct 19 10:42:18 2009 From: wilde at mcs.anl.gov (Michael Wilde) Date: Mon, 19 Oct 2009 10:42:18 -0500 Subject: [Swift-devel] IO overheads of swift wrapper scripts on BlueGene/P In-Reply-To: <1255966481.9399.1.camel@localhost> References: <50b07b4b0910161802p70608e99k354b4b7cccc10b55@mail.gmail.com> <1255743097.16082.17.camel@localhost> <50b07b4b0910161907u2f9a01bdpe2fafe2237d5b4e3@mail.gmail.com> <1255753610.19114.9.camel@localhost> <50b07b4b0910170059h65625594see9aeed8014179c9@mail.gmail.com> <4AD9FF02.6060705@mcs.anl.gov> <50b07b4b0910171957of897b9dy90c349d1069004be@mail.gmail.com> <4ADB20F7.80000@cs.uchicago.edu> <50b07b4b0910180724q630bb22dve489bcad105bd824@mail.gmail.com> <4ADC84DB.6080608@cs.uchicago.edu> <1255966481.9399.1.camel@localhost> Message-ID: <4ADC88DA.8070101@mcs.anl.gov> I'll try to dig into this later today. Allan, are you available to talk by phone/skype? On 10/19/09 10:34 AM, Mihael Hategan wrote: > On Mon, 2009-10-19 at 10:25 -0500, Ioan Raicu wrote: >> OK, I see now. The move in theory should be light weight, right? As >> its just metadata that is changing (e.g. moving on the same >> filesystem, not copying), right? Or is the job dir really in the >> compute node RAM, and the move is actually doing a copy from CN RAM to >> GPFS? > > In Allan's case, I believe it's scratch -> gpfs. > > It's still troubling to see 3 small files taking 3 minutes to be copied. > > From aespinosa at cs.uchicago.edu Mon Oct 19 13:38:58 2009 From: aespinosa at cs.uchicago.edu (Allan Espinosa) Date: Mon, 19 Oct 2009 13:38:58 -0500 Subject: [Swift-devel] IO overheads of swift wrapper scripts on BlueGene/P In-Reply-To: <1255966481.9399.1.camel@localhost> References: <50b07b4b0910161802p70608e99k354b4b7cccc10b55@mail.gmail.com> <50b07b4b0910161907u2f9a01bdpe2fafe2237d5b4e3@mail.gmail.com> <1255753610.19114.9.camel@localhost> <50b07b4b0910170059h65625594see9aeed8014179c9@mail.gmail.com> <4AD9FF02.6060705@mcs.anl.gov> <50b07b4b0910171957of897b9dy90c349d1069004be@mail.gmail.com> <4ADB20F7.80000@cs.uchicago.edu> <50b07b4b0910180724q630bb22dve489bcad105bd824@mail.gmail.com> <4ADC84DB.6080608@cs.uchicago.edu> <1255966481.9399.1.camel@localhost> Message-ID: <50b07b4b0910191138u20e91ca4sd88ad0d424b0e332@mail.gmail.com> right. its from /intrepid-fs0/users/espinosa/scratch to /intrepid-fs0/users/espinosa/persistent I am sure the two are just a single mount point. -allan 2009/10/19 Mihael Hategan : > On Mon, 2009-10-19 at 10:25 -0500, Ioan Raicu wrote: >> OK, I see now. The move in theory should be light weight, right? As >> its just metadata that is changing (e.g. moving on the same >> filesystem, not copying), right? Or is the job dir really in the >> compute node RAM, and the move is actually doing a copy from CN RAM to >> GPFS? > > In Allan's case, I believe it's scratch -> gpfs. > > It's still troubling to see 3 small files taking 3 minutes to be copied. > > > > -- Allan M. Espinosa PhD student, Computer Science University of Chicago From hategan at mcs.anl.gov Mon Oct 19 14:15:22 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 19 Oct 2009 14:15:22 -0500 Subject: [Swift-devel] IO overheads of swift wrapper scripts on BlueGene/P In-Reply-To: <50b07b4b0910191138u20e91ca4sd88ad0d424b0e332@mail.gmail.com> References: <50b07b4b0910161802p70608e99k354b4b7cccc10b55@mail.gmail.com> <50b07b4b0910161907u2f9a01bdpe2fafe2237d5b4e3@mail.gmail.com> <1255753610.19114.9.camel@localhost> <50b07b4b0910170059h65625594see9aeed8014179c9@mail.gmail.com> <4AD9FF02.6060705@mcs.anl.gov> <50b07b4b0910171957of897b9dy90c349d1069004be@mail.gmail.com> <4ADB20F7.80000@cs.uchicago.edu> <50b07b4b0910180724q630bb22dve489bcad105bd824@mail.gmail.com> <4ADC84DB.6080608@cs.uchicago.edu> <1255966481.9399.1.camel@localhost> <50b07b4b0910191138u20e91ca4sd88ad0d424b0e332@mail.gmail.com> Message-ID: <1255979722.14743.5.camel@localhost> On Mon, 2009-10-19 at 13:38 -0500, Allan Espinosa wrote: > right. its from /intrepid-fs0/users/espinosa/scratch to > /intrepid-fs0/users/espinosa/persistent > > I am sure the two are just a single mount point. That would indicate that it's the same filesystem. That's even odder. I think we need to see what exact system calls are the problem. From bugzilla-daemon at mcs.anl.gov Wed Oct 21 14:47:41 2009 From: bugzilla-daemon at mcs.anl.gov (bugzilla-daemon at mcs.anl.gov) Date: Wed, 21 Oct 2009 14:47:41 -0500 (CDT) Subject: [Swift-devel] [Bug 220] New: id's for external data types not stored in rlog for resume Message-ID: https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=220 Summary: id's for external data types not stored in rlog for resume Product: Swift Version: unspecified Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan at mcs.anl.gov ReportedBy: skenny at uchicago.edu when a function returns only an external data type, it seems that no thread id is logged in the .rlog file...instead it contains something like: null.!unmapped for each external. if the user then tries to resume based on that rlog,it goes through 'Initializing' every single job in the workflow and 'finishes successfully' without ever actually picking up where it left off. as a workaround, we are currently writing out files in addition to the externals that are returned. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug. From wilde at mcs.anl.gov Wed Oct 21 15:12:00 2009 From: wilde at mcs.anl.gov (Michael Wilde) Date: Wed, 21 Oct 2009 15:12:00 -0500 Subject: [Swift-devel] Re: [Swift-user] using swift on a cluster In-Reply-To: <70A5AC06FDB5E54482D19E1C04CDFCF30CA39153@BALI.uhd.campus> References: <70A5AC06FDB5E54482D19E1C04CDFCF307C377B3@BALI.uhd.campus><4ADE84D9.6020508@mcs.anl.gov> <70A5AC06FDB5E54482D19E1C04CDFCF307C377B5@BALI.uhd.campus> <70A5AC06FDB5E54482D19E1C04CDFCF307C377B6@BALI.uhd.campus> <4ADEF844.5020202@mcs.anl.gov> <70A5AC06FDB5E54482D19E1C04CDFCF307C377B7@BALI.uhd.campus> <4ADF192B.8020804@mcs.anl.gov> <70A5AC06FDB5E54482D19E1C04CDFCF307C377C0@BALI.uhd.campus> <4ADF46AB.8030603@mcs.anl.gov> <70A5AC06FDB5E54482D19E1C04CDFCF30CA39153@BALI.uhd.campus> Message-ID: <4ADF6B10.2090402@mcs.anl.gov> cool. - Mike On 10/21/09 2:54 PM, Hodgess, Erin wrote: > Yay! > > That was it! > > Thanks, > Erin > > > -----Original Message----- > From: Michael Wilde [mailto:wilde at mcs.anl.gov] > Sent: Wednesday, October 21, 2009 12:37 PM > To: Hodgess, Erin > Cc: Swift User Discussion List > Subject: Re: [Swift-user] using swift on a cluster > > Erin, > > The first line of your sites.xml file seems to be left there in error: > > > [hodgess at grid bin]$ cat sites.xml > > > > Can you remove that and try again? Im not sure how that got parsed. > > - Mike > > On 10/21/09 12:10 PM, Hodgess, Erin wrote: >> Hi again! >> >> Here are the sites.xml and tc.data files. >> >> Thanks, >> Erin >> >> >> [hodgess at grid bin]$ cat sites.xml >> >> >> >> >> >> >> >> /home/hodgess/swiftwork >> .03 >> 10000 >> >> >> >> >> >> /home/hodgess/swiftwork >> .19 >> 10000 >> >> >> >> [hodgess at grid bin]$ cat tc.data >> localhost convert /usr/bin/convert INSTALLED >> INTEL32::LINUX null >> localhost RInvoke /home/hodgess/R-2.9.2/bin/RInvoke.sh >> INSTALLED INTEL32::LINUX null >> condor RInvoke /home/hodgess/R-2.9.2/bin/RInvoke.sh INSTALLED > >> INTEL32::LINUX null >> [hodgess at grid bin]$ cat firstR.R >> cat: firstR.R: No such file or directory >> [hodgess at grid bin]$ cat firstR.swift >> type file{} >> app (file output) firstone (file scriptFile) { >> RInvoke @filename(scriptFile) @filename(output); >> } >> >> >> file scriptFile <"a1.in" >; >> file output <"a1.out" >; >> output=firstone(scriptFile); >> [hodgess at grid bin]$ >> >> >> Erin M. Hodgess, PhD >> Associate Professor >> Department of Computer and Mathematical Sciences >> University of Houston - Downtown >> mailto: hodgesse at uhd.edu >> >> >> >> -----Original Message----- >> From: Michael Wilde [mailto:wilde at mcs.anl.gov] >> Sent: Wed 10/21/2009 9:22 AM >> To: Hodgess, Erin >> Cc: swift-user at ci.uchicago.edu >> Subject: Re: [Swift-user] using swift on a cluster >> >> Erin, we need to look into this further. >> >> Please make sure that you are running either Swift 0.9 or the latest >> source from svn. And tell us what revision you are running. >> >> Also please post your tc.data and sites.xml (and log file is its small >> enought); see if there are any messages in the .log file that would >> clarify the error. >> >> Make sure that your app is cataloged in tc.data as being on pool >> "condor". But I think if it were not, you'd see a different error. >> >> It almost looks to me like Swift is looking for the GRAM service > contact >> string, as if it thinks you are asking for Condor-G instead of local >> Condor, eg: >> >> grid >> > key="gridResource">gt2 > belhaven-1.renci.org/jobmanager-fork >> Just as a test, try changing provider="condor" to "pbs" in sites.xml. > If >> the error changes to something like "PBS not installed" or "qsub not >> found" then I would suspect this is the case. >> >> Its possible you can add just the jobType element with the value set > to >> vanilla instead of grid, but I am purely *guessing*; we'll look deeper >> as soon as you send the info above and we have time. >> >> - Mike >> >> >> On 10/21/09 9:03 AM, Hodgess, Erin wrote: >> > Here is the output: >> > >> > >> > [hodgess at grid bin]$ swift -tc.file tc.data -sites.file sites.xml >> > firstR.swift >> > Swift 0.9 swift-r2860 cog-r2388 >> > >> > RunID: 20091021-0901-aku7y862 >> > Progress: >> > Execution failed: >> > No service contacts available >> > [hodgess at grid bin]$ >> > >> > >> > >> > Erin M. Hodgess, PhD >> > Associate Professor >> > Department of Computer and Mathematical Sciences >> > University of Houston - Downtown >> > mailto: hodgesse at uhd.edu >> > >> > >> > >> > -----Original Message----- >> > From: Michael Wilde [mailto:wilde at mcs.anl.gov] >> > Sent: Wed 10/21/2009 7:02 AM >> > To: Hodgess, Erin >> > Cc: swift-user at ci.uchicago.edu >> > Subject: Re: [Swift-user] using swift on a cluster >> > >> > For running Swift locally on a Condor cluster, use a sites.xml > based on >> > this example: >> > >> > >> > >> > >> > >> > >> > >> > >> > /home/erin/swiftwork >> > .03 >> > key="initialScore">10000 >> > >> > >> > >> > >> > >> > /home/erin/swiftwork >> > .19 >> > key="initialScore">10000 >> > >> > >> > >> > >> > The jobThrottle values above will enable Swift to run up to 4 jobs > at a >> > time on localhost and 20 jobs at a time on the Condor cluster. >> > >> > Use tc.data to catalog applications on pool or the other. >> > >> > Set jobThrottle as desired to control execution parallelism. >> > >> > #jobs run in parallel is (jobThrottle * 100)+1 >> > >> > initialScore=10000 overrides Swift's "start slow" approach to > sensing >> > the site's responsiveness. >> > >> > - Mike >> > >> > On 10/21/09 3:17 AM, Hodgess, Erin wrote: >> > > Aha! >> > > >> > > I needed the universe=vanilla line. >> > > >> > > >> > > >> > > Erin M. Hodgess, PhD >> > > Associate Professor >> > > Department of Computer and Mathematical Sciences >> > > University of Houston - Downtown >> > > mailto: hodgesse at uhd.edu >> > > >> > > >> > > >> > > -----Original Message----- >> > > From: swift-user-bounces at ci.uchicago.edu on behalf of Hodgess, > Erin >> > > Sent: Wed 10/21/2009 3:07 AM >> > > To: Michael Wilde >> > > Cc: swift-user at ci.uchicago.edu >> > > Subject: RE: [Swift-user] using swift on a cluster >> > > >> > > Hello! >> > > >> > > We are indeed using condor. >> > > >> > > I wanted to try a small test run, but am running into trouble: >> > > >> > > [hodgess at grid bin]$ cat myjob.submit >> > > executable=/usr/bin/id >> > > output=results.output >> > > error=results.error >> > > log=results.log >> > > queue >> > > [hodgess at grid bin]$ condor_submit myjob.submit >> > > Submitting job(s). >> > > Logging submit event(s). >> > > 1 job(s) submitted to cluster 15. >> > > [hodgess at grid bin]$ ls results* >> > > results.error results.log results.output >> > > You have new mail in /var/spool/mail/hodgess >> > > [hodgess at grid bin]$ cat results.log >> > > 000 (015.000.000) 10/21 03:06:03 Job submitted from host: >> > > <192.168.1.11:46274> >> > > ... >> > > 001 (015.000.000) 10/21 03:06:05 Job executing on host: >> > <10.1.255.244:44508> >> > > ... >> > > 002 (015.000.000) 10/21 03:06:05 (1) Job not properly linked for > >> Condor. >> > > ... >> > > 009 (015.000.000) 10/21 03:06:05 Job was aborted by the user. >> > > ... >> > > [hodgess at grid bin]$ >> > > >> > > I'm not sure why the job is not linked. >> > > >> > > Any suggestions would be much appreciated. >> > > >> > > Thanks, >> > > Erin >> > > >> > > >> > > Erin M. Hodgess, PhD >> > > Associate Professor >> > > Department of Computer and Mathematical Sciences >> > > University of Houston - Downtown >> > > mailto: hodgesse at uhd.edu >> > > >> > > >> > > >> > > -----Original Message----- >> > > From: Michael Wilde [mailto:wilde at mcs.anl.gov] >> > > Sent: Tue 10/20/2009 10:49 PM >> > > To: Hodgess, Erin >> > > Cc: swift-user at ci.uchicago.edu >> > > Subject: Re: [Swift-user] using swift on a cluster >> > > >> > > Hi Erin, >> > > >> > > I'm assuming you meant "use Swift to run jobs on the compute > nodes of >> > > the cluster"? >> > > >> > > If so, you first need to find out what scheduler (also called > "batch >> > > system" or "local resource manager") the cluster is running. >> > > >> > > Thats typical one of these: PBS, Condor, or SGE. >> > > >> > > Either ask your system administrator, or see if the "man" > command or >> > > similar probes give you a clue: >> > > >> > > Condor: condor_q -version >> > > >> > > condor_q -version >> > > $CondorVersion: 7.2.4 Jun 16 2009 BuildID: 159529 $ >> > > $CondorPlatform: I386-LINUX_RHEL5 $ >> > > >> > > PBS: man qstat: >> > > >> > > qstat(1B) PBS >> > > >> > > SGE: man qstat: >> > > >> > > QSTAT(1) Sun Grid Engine User Commands >> > > >> > > >> > > If its PBS or Condor, then the Swift user guide gives the > sites.xml >> > > entries to use. >> > > >> > > Tell us what you find, then try following the instructions in > the user >> > > guide, and follow up with questions as needed. >> > > >> > > - Mike >> > > >> > > >> > > On 10/20/09 9:41 PM, Hodgess, Erin wrote: >> > > > Hi Swift Users: >> > > > >> > > > I'm on a cluster and would like to use swift on the different > >> sites on >> > > > the cluster. >> > > > >> > > > How would I do that, please? >> > > > >> > > > Thanks, >> > > > Erin >> > > > >> > > > >> > > > Erin M. Hodgess, PhD >> > > > Associate Professor >> > > > Department of Computer and Mathematical Sciences >> > > > University of Houston - Downtown >> > > > mailto: hodgesse at uhd.edu >> > > > >> > > > >> > > > >> > > ------------------------------------------------------------------------ >> > > > >> > > > _______________________________________________ >> > > > Swift-user mailing list >> > > > Swift-user at ci.uchicago.edu >> > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user >> > > >> > > >> > >> > From foster at anl.gov Wed Oct 21 22:02:26 2009 From: foster at anl.gov (Ian Foster) Date: Wed, 21 Oct 2009 22:02:26 -0500 Subject: [Swift-devel] Scala Message-ID: <7149433A-AD43-434B-A2EE-131EADBC0DC9@anl.gov> Could Scala be what we have been looking for? http://www.scala-lang.org/ From wilde at mcs.anl.gov Thu Oct 22 10:56:26 2009 From: wilde at mcs.anl.gov (Michael Wilde) Date: Thu, 22 Oct 2009 10:56:26 -0500 Subject: [Swift-devel] Need gcc 4.4 x86 and x86_64 systems for code building Message-ID: <4AE080AA.9070902@mcs.anl.gov> Hi all, Does anyone have a pair of gcc 4.4-equipped Linux boxes that I can use to build a library on? Thanks, Mike From support at ci.uchicago.edu Fri Oct 23 09:08:21 2009 From: support at ci.uchicago.edu (Mike Wilde) Date: Fri, 23 Oct 2009 09:08:21 -0500 Subject: [Swift-devel] Re: [CI Ticketing System #2091] Need gcc 4.4 x86 and x86_64 systems for code building In-Reply-To: <4AE1B8C9.4090609@mcs.anl.gov> References: <4AE1B8C9.4090609@mcs.anl.gov> Message-ID: You can close this - I found the systems I need. Thanks, Mike On 10/23/09 8:26 AM, Ti Leggett wrote: > This is to inform you that ticket #2091 with subject, Need gcc 4.4 x86 and x86_64 systems for code building, was assigned to Ti Leggett. From skenny at uchicago.edu Fri Oct 23 10:45:04 2009 From: skenny at uchicago.edu (skenny at uchicago.edu) Date: Fri, 23 Oct 2009 10:45:04 -0500 (CDT) Subject: [Swift-devel] burnin' up ranger w/the latest coasters In-Reply-To: <20091014153915.CDW69329@m4500-02.uchicago.edu> References: <20091013111417.CDU59058@m4500-02.uchicago.edu> <20091014153915.CDW69329@m4500-02.uchicago.edu> Message-ID: <20091023104504.CEI95549@m4500-02.uchicago.edu> however...when i use the configs here and i try to run a workflow with 196,608 jobs it seems that coasters starts to ramp up nicely, but maybe a little too well :) as it begins requesting more cores than i'm allowed in the normal queue on ranger. that is, the limit is 4096. i tried changing maxNodes to 4096 which did not work. i'm wondering if workers per node should actually be 16 instead (?) but i know you've gotten it to work well with the setting at 32 so i'm not sure... anyway, it ramped up nicely (and was only like 8 jobs away from finishing the whole thing) i just need to know how to cap it off so it won't ask for more than 4096 cores. thanks ~sk ---- Original message ---- >Date: Wed, 14 Oct 2009 15:39:15 -0500 (CDT) >From: >Subject: Re: [Swift-devel] burnin' up ranger w/the latest coasters >To: swift-user at ci.uchicago.edu, swift-devel at ci.uchicago.edu > >for those interested, here are the config files used for this run: > >swift.properties: > >sites.file=config/coaster_ranger.xml >tc.file=/ci/projects/cnari/config/tc.data >lazy.errors=false >caching.algorithm=LRU >pgraph=false >pgraph.graph.options=splines="compound", rankdir="TB" >pgraph.node.options=color="seagreen", style="filled" >clustering.enabled=false >clustering.queue.delay=4 >clustering.min.time=60 >kickstart.enabled=maybe >kickstart.always.transfer=false >wrapperlog.always.transfer=false >throttle.submit=3 >throttle.host.submit=8 >throttle.score.job.factor=64 >throttle.transfers=16 >throttle.file.operations=16 >sitedir.keep=false >execution.retries=3 >replication.enabled=false >replication.min.queue.time=60 >replication.limit=3 >foreach.max.threads=16384 > >coaster_ranger.xml: > > > > > > > > > key="jobThrottle">1000.0 > url="gt2://gatekeeper.ranger.tacc.teragrid.org"/> > normal > 32 > 1 > 16 > 8192 > 72000 > key="project">TG-DBS080004N > url="gatekeeper.ranger.tacc.teragrid.org" >jobManager="gt2:gt2:SGE"/> > > >/work/00926/tg459516/sidgrid_out/{username} > > > > > >---- Original message ---- >>Date: Tue, 13 Oct 2009 11:14:17 -0500 (CDT) >>From: >>Subject: [Swift-devel] burnin' up ranger w/the latest coasters >>To: swift-devel at ci.uchicago.edu >> >>Final status: Finished successfully:131072 >> >>re-running some of the workflows from our recent SEM >>paper with the latest swift...sadly, queue time on ranger has >>only gone up since those initial runs...but luckily coasters >>has speeded things up, so it ends up evening out for time to >>solution :) >> >>not sure i fully understand the plot: >> >>http://www.ci.uchicago.edu/~skenny/workflows/sem_131k/ >> >>log is here: >> >>/ci/projects/cnari/logs/skenny/4reg_2cond-20091012-1607-ugidm2s2.log >>_______________________________________________ >>Swift-devel mailing list >>Swift-devel at ci.uchicago.edu >>http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel >_______________________________________________ >Swift-devel mailing list >Swift-devel at ci.uchicago.edu >http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel From hategan at mcs.anl.gov Fri Oct 23 11:37:10 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Fri, 23 Oct 2009 11:37:10 -0500 Subject: [Swift-devel] burnin' up ranger w/the latest coasters In-Reply-To: <20091023104504.CEI95549@m4500-02.uchicago.edu> References: <20091013111417.CDU59058@m4500-02.uchicago.edu> <20091014153915.CDW69329@m4500-02.uchicago.edu> <20091023104504.CEI95549@m4500-02.uchicago.edu> Message-ID: <1256315830.10810.5.camel@localhost> On Fri, 2009-10-23 at 10:45 -0500, skenny at uchicago.edu wrote: > however...when i use the configs here and i try to run a > workflow with 196,608 jobs it seems that coasters starts to > ramp up nicely, but maybe a little too well :) as it begins > requesting more cores than i'm allowed in the normal queue on > ranger. that is, the limit is 4096. i tried changing maxNodes > to 4096 which did not work. Shouldn't that be 4096/workersPerNode? > i'm wondering if workers per node > should actually be 16 instead (?) but i know you've gotten it > to work well with the setting at 32 so i'm not sure... You could set it to 16. My reasoning for doubling it was that if the processes you run are slightly I/O bound, then you'd get slightly better performance by running two processes per core. > > anyway, it ramped up nicely (and was only like 8 jobs away > from finishing the whole thing) i just need to know how to cap > it off so it won't ask for more than 4096 cores. > > thanks > ~sk > > ---- Original message ---- > >Date: Wed, 14 Oct 2009 15:39:15 -0500 (CDT) > >From: > >Subject: Re: [Swift-devel] burnin' up ranger w/the latest > coasters > >To: swift-user at ci.uchicago.edu, swift-devel at ci.uchicago.edu > > > >for those interested, here are the config files used for this > run: > > > >swift.properties: > > > >sites.file=config/coaster_ranger.xml > >tc.file=/ci/projects/cnari/config/tc.data > >lazy.errors=false > >caching.algorithm=LRU > >pgraph=false > >pgraph.graph.options=splines="compound", rankdir="TB" > >pgraph.node.options=color="seagreen", style="filled" > >clustering.enabled=false > >clustering.queue.delay=4 > >clustering.min.time=60 > >kickstart.enabled=maybe > >kickstart.always.transfer=false > >wrapperlog.always.transfer=false > >throttle.submit=3 > >throttle.host.submit=8 > >throttle.score.job.factor=64 > >throttle.transfers=16 > >throttle.file.operations=16 > >sitedir.keep=false > >execution.retries=3 > >replication.enabled=false > >replication.min.queue.time=60 > >replication.limit=3 > >foreach.max.threads=16384 > > > >coaster_ranger.xml: > > > > > > > > > > > > > > > > > > >key="jobThrottle">1000.0 > > >url="gt2://gatekeeper.ranger.tacc.teragrid.org"/> > > normal > > 32 > > 1 > > 16 > > 8192 > > 72000 > > >key="project">TG-DBS080004N > > >url="gatekeeper.ranger.tacc.teragrid.org" > >jobManager="gt2:gt2:SGE"/> > > > > > >/work/00926/tg459516/sidgrid_out/{username} > > > > > > > > > > > >---- Original message ---- > >>Date: Tue, 13 Oct 2009 11:14:17 -0500 (CDT) > >>From: > >>Subject: [Swift-devel] burnin' up ranger w/the latest coasters > >>To: swift-devel at ci.uchicago.edu > >> > >>Final status: Finished successfully:131072 > >> > >>re-running some of the workflows from our recent SEM > >>paper with the latest swift...sadly, queue time on ranger has > >>only gone up since those initial runs...but luckily coasters > >>has speeded things up, so it ends up evening out for time to > >>solution :) > >> > >>not sure i fully understand the plot: > >> > >>http://www.ci.uchicago.edu/~skenny/workflows/sem_131k/ > >> > >>log is here: > >> > >>/ci/projects/cnari/logs/skenny/4reg_2cond-20091012-1607-ugidm2s2.log > >>_______________________________________________ > >>Swift-devel mailing list > >>Swift-devel at ci.uchicago.edu > >>http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > >_______________________________________________ > >Swift-devel mailing list > >Swift-devel at ci.uchicago.edu > >http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel From skenny at uchicago.edu Fri Oct 23 13:02:00 2009 From: skenny at uchicago.edu (skenny at uchicago.edu) Date: Fri, 23 Oct 2009 13:02:00 -0500 (CDT) Subject: [Swift-devel] burnin' up ranger w/the latest coasters In-Reply-To: <1256315830.10810.5.camel@localhost> References: <20091013111417.CDU59058@m4500-02.uchicago.edu> <20091014153915.CDW69329@m4500-02.uchicago.edu> <20091023104504.CEI95549@m4500-02.uchicago.edu> <1256315830.10810.5.camel@localhost> Message-ID: <20091023130200.CEJ16713@m4500-02.uchicago.edu> >> however...when i use the configs here and i try to run a >> workflow with 196,608 jobs it seems that coasters starts to >> ramp up nicely, but maybe a little too well :) as it begins >> requesting more cores than i'm allowed in the normal queue on >> ranger. that is, the limit is 4096. i tried changing maxNodes >> to 4096 which did not work. > >Shouldn't that be 4096/workersPerNode? don't think i'm understanding you here...the workersPerNode you originally suggested was 32. why would i increase that to 4096 when what i'm trying to do is request fewer total cores? > >> i'm wondering if workers per node >> should actually be 16 instead (?) but i know you've gotten it >> to work well with the setting at 32 so i'm not sure... > >You could set it to 16. My reasoning for doubling it was that if the >processes you run are slightly I/O bound, then you'd get slightly better >performance by running two processes per core. > >> >> anyway, it ramped up nicely (and was only like 8 jobs away >> from finishing the whole thing) i just need to know how to cap >> it off so it won't ask for more than 4096 cores. >> >> thanks >> ~sk >> >> ---- Original message ---- >> >Date: Wed, 14 Oct 2009 15:39:15 -0500 (CDT) >> >From: >> >Subject: Re: [Swift-devel] burnin' up ranger w/the latest >> coasters >> >To: swift-user at ci.uchicago.edu, swift-devel at ci.uchicago.edu >> > >> >for those interested, here are the config files used for this >> run: >> > >> >swift.properties: >> > >> >sites.file=config/coaster_ranger.xml >> >tc.file=/ci/projects/cnari/config/tc.data >> >lazy.errors=false >> >caching.algorithm=LRU >> >pgraph=false >> >pgraph.graph.options=splines="compound", rankdir="TB" >> >pgraph.node.options=color="seagreen", style="filled" >> >clustering.enabled=false >> >clustering.queue.delay=4 >> >clustering.min.time=60 >> >kickstart.enabled=maybe >> >kickstart.always.transfer=false >> >wrapperlog.always.transfer=false >> >throttle.submit=3 >> >throttle.host.submit=8 >> >throttle.score.job.factor=64 >> >throttle.transfers=16 >> >throttle.file.operations=16 >> >sitedir.keep=false >> >execution.retries=3 >> >replication.enabled=false >> >replication.min.queue.time=60 >> >replication.limit=3 >> >foreach.max.threads=16384 >> > >> >coaster_ranger.xml: >> > >> > >> > >> > >> > >> > >> > >> > >> > > >key="jobThrottle">1000.0 >> > > >url="gt2://gatekeeper.ranger.tacc.teragrid.org"/> >> > normal >> > 32 >> > 1 >> > 16 >> > 8192 >> > 72000 >> > > >key="project">TG-DBS080004N >> > > >url="gatekeeper.ranger.tacc.teragrid.org" >> >jobManager="gt2:gt2:SGE"/> >> > >> > >> >/work/00926/tg459516/sidgrid_out/{username} >> > >> > >> > >> > >> > >> >---- Original message ---- >> >>Date: Tue, 13 Oct 2009 11:14:17 -0500 (CDT) >> >>From: >> >>Subject: [Swift-devel] burnin' up ranger w/the latest coasters >> >>To: swift-devel at ci.uchicago.edu >> >> >> >>Final status: Finished successfully:131072 >> >> >> >>re-running some of the workflows from our recent SEM >> >>paper with the latest swift...sadly, queue time on ranger has >> >>only gone up since those initial runs...but luckily coasters >> >>has speeded things up, so it ends up evening out for time to >> >>solution :) >> >> >> >>not sure i fully understand the plot: >> >> >> >>http://www.ci.uchicago.edu/~skenny/workflows/sem_131k/ >> >> >> >>log is here: >> >> >> >>/ci/projects/cnari/logs/skenny/4reg_2cond-20091012-1607-ugidm2s2.log >> >>_______________________________________________ >> >>Swift-devel mailing list >> >>Swift-devel at ci.uchicago.edu >> >>http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel >> >_______________________________________________ >> >Swift-devel mailing list >> >Swift-devel at ci.uchicago.edu >> >http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel >> _______________________________________________ >> Swift-devel mailing list >> Swift-devel at ci.uchicago.edu >> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > From hategan at mcs.anl.gov Fri Oct 23 13:18:24 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Fri, 23 Oct 2009 13:18:24 -0500 Subject: [Swift-devel] burnin' up ranger w/the latest coasters In-Reply-To: <20091023130200.CEJ16713@m4500-02.uchicago.edu> References: <20091013111417.CDU59058@m4500-02.uchicago.edu> <20091014153915.CDW69329@m4500-02.uchicago.edu> <20091023104504.CEI95549@m4500-02.uchicago.edu> <1256315830.10810.5.camel@localhost> <20091023130200.CEJ16713@m4500-02.uchicago.edu> Message-ID: <1256321904.15412.3.camel@localhost> On Fri, 2009-10-23 at 13:02 -0500, skenny at uchicago.edu wrote: > >> however...when i use the configs here and i try to run a > >> workflow with 196,608 jobs it seems that coasters starts to > >> ramp up nicely, but maybe a little too well :) as it begins > >> requesting more cores than i'm allowed in the normal queue on > >> ranger. that is, the limit is 4096. i tried changing maxNodes > >> to 4096 which did not work. > > > >Shouldn't that be 4096/workersPerNode? > > don't think i'm understanding you here...the workersPerNode > you originally suggested was 32. why would i increase that to > 4096 when what i'm trying to do is request fewer total cores? No. If you have C CORESPerNODE and the maximum number of CORES you can request is 4096, then the maximum NODES you can request is 4096/C not 4096. You are setting maxNodes to 4096. That means it will request 4096*16 cores. From aespinosa at cs.uchicago.edu Fri Oct 23 16:14:41 2009 From: aespinosa at cs.uchicago.edu (Allan Espinosa) Date: Fri, 23 Oct 2009 16:14:41 -0500 Subject: [Swift-devel] Re: IO overheads of swift wrapper scripts on BlueGene/P In-Reply-To: <50b07b4b0910161802p70608e99k354b4b7cccc10b55@mail.gmail.com> References: <50b07b4b0910161802p70608e99k354b4b7cccc10b55@mail.gmail.com> Message-ID: <50b07b4b0910231414v1e5b03a3qf68fde141e2043df@mail.gmail.com> Hello, I just tried Mike's swift build in /home/falkon/swift/src/cog/modules/swift/dist/swift-svn/bin/swift . My workflow ran for 838 seconds. This is pretty comparable to a falkon-only run of 829 seconds. I guess the patches (local CN jobdir) fixes things. thanks Mike! -Allan 2009/10/16 Allan Espinosa : > The attached graph looks interesting. The green on the left shows the > 192 job workload using the straight falkon client. The right one is > through swift. ?Using pure falkon the total time for the workload is > at around 814 seconds while the swift workflow took 1520 seconds. ?In > some repeats of the same run, Swift took 3078 seconds. > From wilde at mcs.anl.gov Fri Oct 23 16:19:49 2009 From: wilde at mcs.anl.gov (Michael Wilde) Date: Fri, 23 Oct 2009 16:19:49 -0500 Subject: [Swift-devel] Re: IO overheads of swift wrapper scripts on BlueGene/P In-Reply-To: <50b07b4b0910231414v1e5b03a3qf68fde141e2043df@mail.gmail.com> References: <50b07b4b0910161802p70608e99k354b4b7cccc10b55@mail.gmail.com> <50b07b4b0910231414v1e5b03a3qf68fde141e2043df@mail.gmail.com> Message-ID: <4AE21DF5.2020006@mcs.anl.gov> Cool, Allan, thanks for the update. Especially for our CDM work, it would be useful to go back and figure out where the lengthy delays were coming from, as that helps justify CDM. I suspect its contention in the creation of jobdirs on shared storage, or contention within a jobdir, or both. We'll want to see how this overhead grows with scale; it doesnt seem to take much parallelism to cause bad degradation with the default data management. - Mike On 10/23/09 4:14 PM, Allan Espinosa wrote: > Hello, > > I just tried Mike's swift build in > /home/falkon/swift/src/cog/modules/swift/dist/swift-svn/bin/swift . > My workflow ran for 838 seconds. This is pretty comparable to a > falkon-only run of 829 seconds. > > I guess the patches (local CN jobdir) fixes things. > > thanks Mike! > > -Allan > > 2009/10/16 Allan Espinosa : >> The attached graph looks interesting. The green on the left shows the >> 192 job workload using the straight falkon client. The right one is >> through swift. Using pure falkon the total time for the workload is >> at around 814 seconds while the swift workflow took 1520 seconds. In >> some repeats of the same run, Swift took 3078 seconds. >> From hategan at mcs.anl.gov Fri Oct 23 16:27:28 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Fri, 23 Oct 2009 16:27:28 -0500 Subject: [Swift-devel] Re: IO overheads of swift wrapper scripts on BlueGene/P In-Reply-To: <50b07b4b0910231414v1e5b03a3qf68fde141e2043df@mail.gmail.com> References: <50b07b4b0910161802p70608e99k354b4b7cccc10b55@mail.gmail.com> <50b07b4b0910231414v1e5b03a3qf68fde141e2043df@mail.gmail.com> Message-ID: <1256333248.19268.0.camel@localhost> I'd still want to understand what went wrong. On Fri, 2009-10-23 at 16:14 -0500, Allan Espinosa wrote: > Hello, > > I just tried Mike's swift build in > /home/falkon/swift/src/cog/modules/swift/dist/swift-svn/bin/swift . > My workflow ran for 838 seconds. This is pretty comparable to a > falkon-only run of 829 seconds. > > I guess the patches (local CN jobdir) fixes things. > > thanks Mike! > > -Allan > > 2009/10/16 Allan Espinosa : > > The attached graph looks interesting. The green on the left shows the > > 192 job workload using the straight falkon client. The right one is > > through swift. Using pure falkon the total time for the workload is > > at around 814 seconds while the swift workflow took 1520 seconds. In > > some repeats of the same run, Swift took 3078 seconds. > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel From benc at hawaga.org.uk Sat Oct 24 00:23:23 2009 From: benc at hawaga.org.uk (Ben Clifford) Date: Sat, 24 Oct 2009 05:23:23 +0000 (GMT) Subject: [Swift-devel] Scala In-Reply-To: <7149433A-AD43-434B-A2EE-131EADBC0DC9@anl.gov> References: <7149433A-AD43-434B-A2EE-131EADBC0DC9@anl.gov> Message-ID: On Wed, 21 Oct 2009, Ian Foster wrote: > Could Scala be what we have been looking for? > > http://www.scala-lang.org/ The thing that seems hard to get from any language, and thus drives swiftscript to be its own language, is the everything-parallel execution of expressions. Nothing that I've seen in scala (which admittedly I haven't thought about much) seems revolutionary in that respect. -- From foster at anl.gov Sat Oct 24 19:51:53 2009 From: foster at anl.gov (Ian Foster) Date: Sat, 24 Oct 2009 19:51:53 -0500 Subject: [Swift-devel] Scala In-Reply-To: References: <7149433A-AD43-434B-A2EE-131EADBC0DC9@anl.gov> Message-ID: Hi Ben: I gather you are in Australia at present? Say hello from me. I think that the key feature of Swift (apart from functional semantics and XDTM-like mechanisms) is its use of single assignment variables for synchronization. I'm not sure that one has to have everything-parallel semantics to do what we want. One could also have explicitly parallel operators, like "forall" and "par" used in CC++. I remain interested in the question of whether we can reduce barriers to use of Swift, and its long-term maintenance costs, by adopting some existing technology to build on. Scala is one of several potential candidates. Ian. On Oct 24, 2009, at 12:23 AM, Ben Clifford wrote: > > On Wed, 21 Oct 2009, Ian Foster wrote: > >> Could Scala be what we have been looking for? >> >> http://www.scala-lang.org/ > > The thing that seems hard to get from any language, and thus drives > swiftscript to be its own language, is the everything-parallel > execution > of expressions. Nothing that I've seen in scala (which admittedly I > haven't thought about much) seems revolutionary in that respect. > > -- From hategan at mcs.anl.gov Sat Oct 24 21:07:22 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sat, 24 Oct 2009 21:07:22 -0500 Subject: [Swift-devel] Scala In-Reply-To: References: <7149433A-AD43-434B-A2EE-131EADBC0DC9@anl.gov> Message-ID: <1256436442.11025.67.camel@localhost> On Sat, 2009-10-24 at 19:51 -0500, Ian Foster wrote: > Hi Ben: > > I gather you are in Australia at present? Say hello from me. > > I think that the key feature of Swift (apart from functional semantics > and XDTM-like mechanisms) is its use of single assignment variables > for synchronization. > > I'm not sure that one has to have everything-parallel semantics to do > what we want. One could also have explicitly parallel operators, like > "forall" and "par" used in CC++. We could have forall and par in Java, as libraries. Which is something I would personally like to see regardless of what Swift does. I'm not sure why Scala would be better (other than being a more interesting but more obscure language - which is a milder version of the Swift - Java conflict). > > I remain interested in the question of whether we can reduce barriers > to use of Swift, and its long-term maintenance costs, by adopting some > existing technology to build on. Scala is one of several potential > candidates. My opinion on that is that what we're trying to address with Swift (as a language) is for us to deal with the nasty problems of concurrency so that our users don't have to (questions of whether we succeed at that notwithstanding). Choosing Scala or Java instead would mean that, while being able to work with a more stable and better known language (as well as some of us having an easier job), our users will have to take care of some of those harder problems. Taking the limit, the maintenance costs of doing nothing are pretty low. From wilde at mcs.anl.gov Sun Oct 25 11:00:17 2009 From: wilde at mcs.anl.gov (Michael Wilde) Date: Sun, 25 Oct 2009 11:00:17 -0500 Subject: [Swift-devel] Scala In-Reply-To: <1256436442.11025.67.camel@localhost> References: <7149433A-AD43-434B-A2EE-131EADBC0DC9@anl.gov> <1256436442.11025.67.camel@localhost> Message-ID: <4AE47611.1030705@mcs.anl.gov> I believe rather than examining more exotic languages (which themselves are likely struggling to gain a toehold) we should stick to and rally behind the earlier idea of expressing "swift" semantics as a library, which can be used from the already popular languages: Python, Java (+Groovy et al), Perl, Ruby - in roughly that order. Probably C/C++ as well. We should continue, I feel, to pursue two paths: - improve Swift itself, especially in documentation and support tools, because it works and is gaining in usage, from which we can gain a lot of experience. Swift's slow pace adoption step more from documentation/tutorial shortcomings, rough edges in environment support and debugging, and lack of people to do these things, than from an intrinsic lack of market attractiveness. Swift is a thin language by design, which should enhance its attractiveness. - try some simple experiments with a Swift library for Python, to see how it feels to program in, and how much of our current runtime infrastructure we can reuse. Swift-as-a-library seems to me to have potential similar to MPI to achieve great success - wider that Swift-as-a-language - while not eliminating the benefits and continued support and improvement of the current system. A Swift function library could have these elements: - declare data types - declare data(set) instances - define app wrappers - define blocks (dataflow dags) - define foreach-iterators - launch a script - monitor script progress and extract results It should be possible using these primitives to both run the semantics of most or all current swift programs, and likely to re-use, at least initially, large portions of the Swift runtime system. For example, if we were to do an experiment in Jython, we could embed much of swift and the karajan engine into a single process with the Python language iterpreter. We could also implement the Swift engine as a companion process with a socket interface to accept and process definitions and script execution requests, and this engine could both continue to serve the current Swift execution base and as a the engine for other language bindings. - Mike On 10/24/09 9:07 PM, Mihael Hategan wrote: > On Sat, 2009-10-24 at 19:51 -0500, Ian Foster wrote: >> Hi Ben: >> >> I gather you are in Australia at present? Say hello from me. >> >> I think that the key feature of Swift (apart from functional semantics >> and XDTM-like mechanisms) is its use of single assignment variables >> for synchronization. >> >> I'm not sure that one has to have everything-parallel semantics to do >> what we want. One could also have explicitly parallel operators, like >> "forall" and "par" used in CC++. > > We could have forall and par in Java, as libraries. Which is something I > would personally like to see regardless of what Swift does. I'm not sure > why Scala would be better (other than being a more interesting but more > obscure language - which is a milder version of the Swift - Java > conflict). > >> I remain interested in the question of whether we can reduce barriers >> to use of Swift, and its long-term maintenance costs, by adopting some >> existing technology to build on. Scala is one of several potential >> candidates. > > My opinion on that is that what we're trying to address with Swift (as a > language) is for us to deal with the nasty problems of concurrency so > that our users don't have to (questions of whether we succeed at that > notwithstanding). > > Choosing Scala or Java instead would mean that, while being able to work > with a more stable and better known language (as well as some of us > having an easier job), our users will have to take care of some of those > harder problems. > > Taking the limit, the maintenance costs of doing nothing are pretty low. > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel From hategan at mcs.anl.gov Sun Oct 25 11:56:38 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sun, 25 Oct 2009 11:56:38 -0500 Subject: [Swift-devel] Scala In-Reply-To: <4AE47611.1030705@mcs.anl.gov> References: <7149433A-AD43-434B-A2EE-131EADBC0DC9@anl.gov> <1256436442.11025.67.camel@localhost> <4AE47611.1030705@mcs.anl.gov> Message-ID: <1256489798.21416.4.camel@localhost> Do we have funding for this or is this going to be yet another way to ensure that what we started with isn't going to get done? On Sun, 2009-10-25 at 11:00 -0500, Michael Wilde wrote: > I believe rather than examining more exotic languages (which themselves > are likely struggling to gain a toehold) we should stick to and rally > behind the earlier idea of expressing "swift" semantics as a library, > which can be used from the already popular languages: Python, Java > (+Groovy et al), Perl, Ruby - in roughly that order. Probably C/C++ as well. > > We should continue, I feel, to pursue two paths: > > - improve Swift itself, especially in documentation and support tools, > because it works and is gaining in usage, from which we can gain a lot > of experience. Swift's slow pace adoption step more from > documentation/tutorial shortcomings, rough edges in environment support > and debugging, and lack of people to do these things, than from an > intrinsic lack of market attractiveness. Swift is a thin language by > design, which should enhance its attractiveness. > > - try some simple experiments with a Swift library for Python, to see > how it feels to program in, and how much of our current runtime > infrastructure we can reuse. > > Swift-as-a-library seems to me to have potential similar to MPI to > achieve great success - wider that Swift-as-a-language - while not > eliminating the benefits and continued support and improvement of the > current system. > > A Swift function library could have these elements: > > - declare data types > - declare data(set) instances > - define app wrappers > - define blocks (dataflow dags) > - define foreach-iterators > - launch a script > - monitor script progress and extract results > > It should be possible using these primitives to both run the semantics > of most or all current swift programs, and likely to re-use, at least > initially, large portions of the Swift runtime system. > > For example, if we were to do an experiment in Jython, we could embed > much of swift and the karajan engine into a single process with the > Python language iterpreter. > > We could also implement the Swift engine as a companion process with a > socket interface to accept and process definitions and script execution > requests, and this engine could both continue to serve the current Swift > execution base and as a the engine for other language bindings. > > - Mike > > > > On 10/24/09 9:07 PM, Mihael Hategan wrote: > > On Sat, 2009-10-24 at 19:51 -0500, Ian Foster wrote: > >> Hi Ben: > >> > >> I gather you are in Australia at present? Say hello from me. > >> > >> I think that the key feature of Swift (apart from functional semantics > >> and XDTM-like mechanisms) is its use of single assignment variables > >> for synchronization. > >> > >> I'm not sure that one has to have everything-parallel semantics to do > >> what we want. One could also have explicitly parallel operators, like > >> "forall" and "par" used in CC++. > > > > We could have forall and par in Java, as libraries. Which is something I > > would personally like to see regardless of what Swift does. I'm not sure > > why Scala would be better (other than being a more interesting but more > > obscure language - which is a milder version of the Swift - Java > > conflict). > > > >> I remain interested in the question of whether we can reduce barriers > >> to use of Swift, and its long-term maintenance costs, by adopting some > >> existing technology to build on. Scala is one of several potential > >> candidates. > > > > My opinion on that is that what we're trying to address with Swift (as a > > language) is for us to deal with the nasty problems of concurrency so > > that our users don't have to (questions of whether we succeed at that > > notwithstanding). > > > > Choosing Scala or Java instead would mean that, while being able to work > > with a more stable and better known language (as well as some of us > > having an easier job), our users will have to take care of some of those > > harder problems. > > > > Taking the limit, the maintenance costs of doing nothing are pretty low. > > > > > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel From wilde at mcs.anl.gov Sun Oct 25 12:30:37 2009 From: wilde at mcs.anl.gov (Michael Wilde) Date: Sun, 25 Oct 2009 12:30:37 -0500 Subject: [Swift-devel] Scala In-Reply-To: <1256489798.21416.4.camel@localhost> References: <7149433A-AD43-434B-A2EE-131EADBC0DC9@anl.gov> <1256436442.11025.67.camel@localhost> <4AE47611.1030705@mcs.anl.gov> <1256489798.21416.4.camel@localhost> Message-ID: <4AE48B3D.9040802@mcs.anl.gov> Here's my thinking. When I said "2 paths", I meant one path that we are executing and another path that we are thinking about. It still does seem reasonable to think once in a while :) But we should stay focused on the first path - and I believe we are. We should keep discussing the second path, as a side-conversation geared to future work and future funding, which this discussion is. Once we have some ideas for a library more distilled, we should carve out a small finite time to try it and see how it feels. If it looks like the library (from gedanken experiments) would be pleasing to code in, we should see if we can find a cheap way to prototype it and see if its feasible. That could lead to a paper and then a proposal (and then funding). But the main point of my posting below is this: as you and Ben point out, most other parallel languages are geared toward parallel threads of control for tasks that are geared mostly to in-memory operations. The Swift execution model is geared to parallel execution of external applications and and data movement. That's unlikely to be found in any other language, and hence argues for either a language focused on that model (e.g., Swift) or a library that can implement that model, where the host language is used as "glue" between operations of the model. My main point is that if we want a good host language, lets focus on popular languages rather than esoteric ones. - Mike On 10/25/09 11:56 AM, Mihael Hategan wrote: > Do we have funding for this or is this going to be yet another way to > ensure that what we started with isn't going to get done? > > On Sun, 2009-10-25 at 11:00 -0500, Michael Wilde wrote: >> I believe rather than examining more exotic languages (which themselves >> are likely struggling to gain a toehold) we should stick to and rally >> behind the earlier idea of expressing "swift" semantics as a library, >> which can be used from the already popular languages: Python, Java >> (+Groovy et al), Perl, Ruby - in roughly that order. Probably C/C++ as well. >> >> We should continue, I feel, to pursue two paths: >> >> - improve Swift itself, especially in documentation and support tools, >> because it works and is gaining in usage, from which we can gain a lot >> of experience. Swift's slow pace adoption step more from >> documentation/tutorial shortcomings, rough edges in environment support >> and debugging, and lack of people to do these things, than from an >> intrinsic lack of market attractiveness. Swift is a thin language by >> design, which should enhance its attractiveness. >> >> - try some simple experiments with a Swift library for Python, to see >> how it feels to program in, and how much of our current runtime >> infrastructure we can reuse. >> >> Swift-as-a-library seems to me to have potential similar to MPI to >> achieve great success - wider that Swift-as-a-language - while not >> eliminating the benefits and continued support and improvement of the >> current system. >> >> A Swift function library could have these elements: >> >> - declare data types >> - declare data(set) instances >> - define app wrappers >> - define blocks (dataflow dags) >> - define foreach-iterators >> - launch a script >> - monitor script progress and extract results >> >> It should be possible using these primitives to both run the semantics >> of most or all current swift programs, and likely to re-use, at least >> initially, large portions of the Swift runtime system. >> >> For example, if we were to do an experiment in Jython, we could embed >> much of swift and the karajan engine into a single process with the >> Python language iterpreter. >> >> We could also implement the Swift engine as a companion process with a >> socket interface to accept and process definitions and script execution >> requests, and this engine could both continue to serve the current Swift >> execution base and as a the engine for other language bindings. >> >> - Mike >> >> >> >> On 10/24/09 9:07 PM, Mihael Hategan wrote: >>> On Sat, 2009-10-24 at 19:51 -0500, Ian Foster wrote: >>>> Hi Ben: >>>> >>>> I gather you are in Australia at present? Say hello from me. >>>> >>>> I think that the key feature of Swift (apart from functional semantics >>>> and XDTM-like mechanisms) is its use of single assignment variables >>>> for synchronization. >>>> >>>> I'm not sure that one has to have everything-parallel semantics to do >>>> what we want. One could also have explicitly parallel operators, like >>>> "forall" and "par" used in CC++. >>> We could have forall and par in Java, as libraries. Which is something I >>> would personally like to see regardless of what Swift does. I'm not sure >>> why Scala would be better (other than being a more interesting but more >>> obscure language - which is a milder version of the Swift - Java >>> conflict). >>> >>>> I remain interested in the question of whether we can reduce barriers >>>> to use of Swift, and its long-term maintenance costs, by adopting some >>>> existing technology to build on. Scala is one of several potential >>>> candidates. >>> My opinion on that is that what we're trying to address with Swift (as a >>> language) is for us to deal with the nasty problems of concurrency so >>> that our users don't have to (questions of whether we succeed at that >>> notwithstanding). >>> >>> Choosing Scala or Java instead would mean that, while being able to work >>> with a more stable and better known language (as well as some of us >>> having an easier job), our users will have to take care of some of those >>> harder problems. >>> >>> Taking the limit, the maintenance costs of doing nothing are pretty low. >>> >>> >>> _______________________________________________ >>> Swift-devel mailing list >>> Swift-devel at ci.uchicago.edu >>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > From hategan at mcs.anl.gov Sun Oct 25 12:31:52 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sun, 25 Oct 2009 12:31:52 -0500 Subject: [Swift-devel] Scala In-Reply-To: <4AE47611.1030705@mcs.anl.gov> References: <7149433A-AD43-434B-A2EE-131EADBC0DC9@anl.gov> <1256436442.11025.67.camel@localhost> <4AE47611.1030705@mcs.anl.gov> Message-ID: <1256491912.21991.37.camel@localhost> On Sun, 2009-10-25 at 11:00 -0500, Michael Wilde wrote: > Swift-as-a-library seems to me to have potential similar to MPI to > achieve great success - wider that Swift-as-a-language - while not > eliminating the benefits and continued support and improvement of the > current system. Btw, I have reasons to believe this is not true. CoG+Java (and then the python cog + python) already provide the tools necessary to achieve most of this. A parallel parameter sweep would look like this (boilerplate removed): Task[] tasks = new Task[params.length]; for(int i = 0; i < params.length; i++) { tasks[i] = buildAndSubmitTask(params[i]); } for(int i = 0; i < tasks.length; i++) { tasks[i].waitFor(); } There is also a scheduler to take care of scheduling, with an API and many other things. All documented, with examples, a hefty FAQ, papers, posters, etc. Didn't work. I suspect because complexity is not in how hard it is to do something, but how hard it is to distinguish the right solution from the sea of potential solutions. In which case a "canned" solution works better. And frankly I wouldn't use it either. If something goes wrong from a programming standpoint, throubleshooting each "workflow" is a pain, because there is no clear separation between the domain specific problem and the library. In debugging, I wouldn't be able to separate the two. Whereas in Swift as it is, it's much easier to distinguish between a runtime-swift bug and having written your script wrong. The library is a carrot in a soup. Swift is layered. From hategan at mcs.anl.gov Sun Oct 25 12:33:47 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sun, 25 Oct 2009 12:33:47 -0500 Subject: [Swift-devel] Scala In-Reply-To: <4AE48B3D.9040802@mcs.anl.gov> References: <7149433A-AD43-434B-A2EE-131EADBC0DC9@anl.gov> <1256436442.11025.67.camel@localhost> <4AE47611.1030705@mcs.anl.gov> <1256489798.21416.4.camel@localhost> <4AE48B3D.9040802@mcs.anl.gov> Message-ID: <1256492027.21991.39.camel@localhost> On Sun, 2009-10-25 at 12:30 -0500, Michael Wilde wrote: > Here's my thinking. When I said "2 paths", I meant one path that we are > executing and another path that we are thinking about. It still does > seem reasonable to think once in a while :) Sorry. I misread what you wrote. > > But we should stay focused on the first path - and I believe we are. I agree entirely. From wilde at mcs.anl.gov Sun Oct 25 12:37:42 2009 From: wilde at mcs.anl.gov (Michael Wilde) Date: Sun, 25 Oct 2009 12:37:42 -0500 Subject: [Swift-devel] Scala In-Reply-To: <1256491912.21991.37.camel@localhost> References: <7149433A-AD43-434B-A2EE-131EADBC0DC9@anl.gov> <1256436442.11025.67.camel@localhost> <4AE47611.1030705@mcs.anl.gov> <1256491912.21991.37.camel@localhost> Message-ID: <4AE48CE6.9040606@mcs.anl.gov> You make some good points here Mihael. Im going to drop off this thread while I mull it over further. There is some user demand from the R/OpenMx community to also pursue an host-language embedding of Swift semantics. That might be a good place to explore this (which *does* have funding). R has the library packages "Snow" and "Snowfall" which do something similar and have proven useful and popular - perhaps a good model to examine and perhaps integrate Swift into. - Mike On 10/25/09 12:31 PM, Mihael Hategan wrote: > On Sun, 2009-10-25 at 11:00 -0500, Michael Wilde wrote: > >> Swift-as-a-library seems to me to have potential similar to MPI to >> achieve great success - wider that Swift-as-a-language - while not >> eliminating the benefits and continued support and improvement of the >> current system. > > Btw, I have reasons to believe this is not true. CoG+Java (and then the > python cog + python) already provide the tools necessary to achieve most > of this. A parallel parameter sweep would look like this (boilerplate > removed): > > Task[] tasks = new Task[params.length]; > for(int i = 0; i < params.length; i++) { > tasks[i] = buildAndSubmitTask(params[i]); > } > for(int i = 0; i < tasks.length; i++) { > tasks[i].waitFor(); > } > > There is also a scheduler to take care of scheduling, with an API and > many other things. All documented, with examples, a hefty FAQ, papers, > posters, etc. > > Didn't work. I suspect because complexity is not in how hard it is to do > something, but how hard it is to distinguish the right solution from the > sea of potential solutions. In which case a "canned" solution works > better. > > And frankly I wouldn't use it either. If something goes wrong from a > programming standpoint, throubleshooting each "workflow" is a pain, > because there is no clear separation between the domain specific problem > and the library. In debugging, I wouldn't be able to separate the two. > Whereas in Swift as it is, it's much easier to distinguish between a > runtime-swift bug and having written your script wrong. The library is a > carrot in a soup. Swift is layered. > From aespinosa at cs.uchicago.edu Sun Oct 25 15:13:50 2009 From: aespinosa at cs.uchicago.edu (Allan Espinosa) Date: Sun, 25 Oct 2009 15:13:50 -0500 Subject: [Swift-devel] Scala In-Reply-To: <1256491912.21991.37.camel@localhost> References: <7149433A-AD43-434B-A2EE-131EADBC0DC9@anl.gov> <1256436442.11025.67.camel@localhost> <4AE47611.1030705@mcs.anl.gov> <1256491912.21991.37.camel@localhost> Message-ID: <50b07b4b0910251313i4fcb7766wd90f4c3a13412011@mail.gmail.com> Also, popular scripting languages python (jython), ruby(jruby), perl( jerl???) can import java classes in their equivalent java implementations. We can play around calling CoG+ java classes to make the workflows :) 2009/10/25 Mihael Hategan : > On Sun, 2009-10-25 at 11:00 -0500, Michael Wilde wrote: > >> Swift-as-a-library seems to me to have potential similar to MPI to >> achieve great success - wider that Swift-as-a-language - while not >> eliminating the benefits and continued support and improvement of the >> current system. > > Btw, I have reasons to believe this is not true. CoG+Java (and then the > python cog + python) already provide the tools necessary to achieve most > of this. A parallel parameter sweep would look like this (boilerplate > removed): > > Task[] tasks = new Task[params.length]; > for(int i = 0; i < params.length; i++) { > ?tasks[i] = buildAndSubmitTask(params[i]); > } > for(int i = 0; i < tasks.length; i++) { > ?tasks[i].waitFor(); > } > > There is also a scheduler to take care of scheduling, with an API and > many other things. All documented, with examples, a hefty FAQ, papers, > posters, etc. > > Didn't work. I suspect because complexity is not in how hard it is to do > something, but how hard it is to distinguish the right solution from the > sea of potential solutions. In which case a "canned" solution works > better. > > And frankly I wouldn't use it either. If something goes wrong from a > programming standpoint, throubleshooting each "workflow" is a pain, > because there is no clear separation between the domain specific problem > and the library. In debugging, I wouldn't be able to separate the two. > Whereas in Swift as it is, it's much easier to distinguish between a > runtime-swift bug and having written your script wrong. The library is a > carrot in a soup. Swift is layered. > From hategan at mcs.anl.gov Sun Oct 25 15:22:40 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sun, 25 Oct 2009 15:22:40 -0500 Subject: [Swift-devel] Scala In-Reply-To: <50b07b4b0910251313i4fcb7766wd90f4c3a13412011@mail.gmail.com> References: <7149433A-AD43-434B-A2EE-131EADBC0DC9@anl.gov> <1256436442.11025.67.camel@localhost> <4AE47611.1030705@mcs.anl.gov> <1256491912.21991.37.camel@localhost> <50b07b4b0910251313i4fcb7766wd90f4c3a13412011@mail.gmail.com> Message-ID: <1256502160.27333.0.camel@localhost> On Sun, 2009-10-25 at 15:13 -0500, Allan Espinosa wrote: > perl( > jerl???) I don't think anybody would dare implement that one :) From wilde at mcs.anl.gov Sun Oct 25 16:33:58 2009 From: wilde at mcs.anl.gov (Michael Wilde) Date: Sun, 25 Oct 2009 16:33:58 -0500 Subject: [Swift-devel] Scala In-Reply-To: <70A5AC06FDB5E54482D19E1C04CDFCF307C377E8@BALI.uhd.campus> References: <7149433A-AD43-434B-A2EE-131EADBC0DC9@anl.gov> <1256436442.11025.67.camel@localhost><4AE47611.1030705@mcs.anl.gov><1256491912.21991.37.camel@localhost> <4AE48CE6.9040606@mcs.anl.gov> <70A5AC06FDB5E54482D19E1C04CDFCF307C377E8@BALI.uhd.campus> Message-ID: <4AE4C446.4090905@mcs.anl.gov> Yes, the OpenMx project might have a use for that, Erin. We'll be developing more Swift interfaces to it first; Rmpi might be needed later if the need comes up for finer-grained parallelism. - Mike On 10/25/09 4:23 PM, Hodgess, Erin wrote: > Would you be interested in Rmpi, by any chance, please? > > I'm doing a little bit of work on that at the moment. > > Thanks, > Erin > > > Erin M. Hodgess, PhD > Associate Professor > Department of Computer and Mathematical Sciences > University of Houston - Downtown > mailto: hodgesse at uhd.edu > > > > -----Original Message----- > From: swift-devel-bounces at ci.uchicago.edu on behalf of Michael Wilde > Sent: Sun 10/25/2009 12:37 PM > To: Mihael Hategan > Cc: swift-devel; Ian Foster > Subject: Re: [Swift-devel] Scala > > You make some good points here Mihael. Im going to drop off this thread > while I mull it over further. There is some user demand from the > R/OpenMx community to also pursue an host-language embedding of Swift > semantics. That might be a good place to explore this (which *does* have > funding). R has the library packages "Snow" and "Snowfall" which do > something similar and have proven useful and popular - perhaps a good > model to examine and perhaps integrate Swift into. > > - Mike > > > On 10/25/09 12:31 PM, Mihael Hategan wrote: > > On Sun, 2009-10-25 at 11:00 -0500, Michael Wilde wrote: > > > >> Swift-as-a-library seems to me to have potential similar to MPI to > >> achieve great success - wider that Swift-as-a-language - while not > >> eliminating the benefits and continued support and improvement of the > >> current system. > > > > Btw, I have reasons to believe this is not true. CoG+Java (and then the > > python cog + python) already provide the tools necessary to achieve most > > of this. A parallel parameter sweep would look like this (boilerplate > > removed): > > > > Task[] tasks = new Task[params.length]; > > for(int i = 0; i < params.length; i++) { > > tasks[i] = buildAndSubmitTask(params[i]); > > } > > for(int i = 0; i < tasks.length; i++) { > > tasks[i].waitFor(); > > } > > > > There is also a scheduler to take care of scheduling, with an API and > > many other things. All documented, with examples, a hefty FAQ, papers, > > posters, etc. > > > > Didn't work. I suspect because complexity is not in how hard it is to do > > something, but how hard it is to distinguish the right solution from the > > sea of potential solutions. In which case a "canned" solution works > > better. > > > > And frankly I wouldn't use it either. If something goes wrong from a > > programming standpoint, throubleshooting each "workflow" is a pain, > > because there is no clear separation between the domain specific problem > > and the library. In debugging, I wouldn't be able to separate the two. > > Whereas in Swift as it is, it's much easier to distinguish between a > > runtime-swift bug and having written your script wrong. The library is a > > carrot in a soup. Swift is layered. > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > From HodgessE at uhd.edu Sun Oct 25 16:23:52 2009 From: HodgessE at uhd.edu (Hodgess, Erin) Date: Sun, 25 Oct 2009 16:23:52 -0500 Subject: [Swift-devel] Scala References: <7149433A-AD43-434B-A2EE-131EADBC0DC9@anl.gov> <1256436442.11025.67.camel@localhost><4AE47611.1030705@mcs.anl.gov><1256491912.21991.37.camel@localhost> <4AE48CE6.9040606@mcs.anl.gov> Message-ID: <70A5AC06FDB5E54482D19E1C04CDFCF307C377E8@BALI.uhd.campus> Would you be interested in Rmpi, by any chance, please? I'm doing a little bit of work on that at the moment. Thanks, Erin Erin M. Hodgess, PhD Associate Professor Department of Computer and Mathematical Sciences University of Houston - Downtown mailto: hodgesse at uhd.edu -----Original Message----- From: swift-devel-bounces at ci.uchicago.edu on behalf of Michael Wilde Sent: Sun 10/25/2009 12:37 PM To: Mihael Hategan Cc: swift-devel; Ian Foster Subject: Re: [Swift-devel] Scala You make some good points here Mihael. Im going to drop off this thread while I mull it over further. There is some user demand from the R/OpenMx community to also pursue an host-language embedding of Swift semantics. That might be a good place to explore this (which *does* have funding). R has the library packages "Snow" and "Snowfall" which do something similar and have proven useful and popular - perhaps a good model to examine and perhaps integrate Swift into. - Mike On 10/25/09 12:31 PM, Mihael Hategan wrote: > On Sun, 2009-10-25 at 11:00 -0500, Michael Wilde wrote: > >> Swift-as-a-library seems to me to have potential similar to MPI to >> achieve great success - wider that Swift-as-a-language - while not >> eliminating the benefits and continued support and improvement of the >> current system. > > Btw, I have reasons to believe this is not true. CoG+Java (and then the > python cog + python) already provide the tools necessary to achieve most > of this. A parallel parameter sweep would look like this (boilerplate > removed): > > Task[] tasks = new Task[params.length]; > for(int i = 0; i < params.length; i++) { > tasks[i] = buildAndSubmitTask(params[i]); > } > for(int i = 0; i < tasks.length; i++) { > tasks[i].waitFor(); > } > > There is also a scheduler to take care of scheduling, with an API and > many other things. All documented, with examples, a hefty FAQ, papers, > posters, etc. > > Didn't work. I suspect because complexity is not in how hard it is to do > something, but how hard it is to distinguish the right solution from the > sea of potential solutions. In which case a "canned" solution works > better. > > And frankly I wouldn't use it either. If something goes wrong from a > programming standpoint, throubleshooting each "workflow" is a pain, > because there is no clear separation between the domain specific problem > and the library. In debugging, I wouldn't be able to separate the two. > Whereas in Swift as it is, it's much easier to distinguish between a > runtime-swift bug and having written your script wrong. The library is a > carrot in a soup. Swift is layered. > _______________________________________________ Swift-devel mailing list Swift-devel at ci.uchicago.edu http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Mon Oct 26 11:56:49 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 26 Oct 2009 11:56:49 -0500 Subject: [Swift-devel] Swift and BGP Message-ID: <1256576209.1135.46.camel@localhost> I've been playing with Swift on the BGP the past few days. My observation is that with the current swift and reasonable mapping (such as the one done by the concurrent mapper) filesystem slowness does not seem to be caused by contentious access to the same file/directory. Instead, it's the sheer amount of network filesystem requests which come from a few sources: - bash: every time bash forks a process it closes the current script, forks the process; after the process is done, bash re-opens the script file, seeks to the position it left it at and reads the next command. And typical scripts involve forking a few processes. Our wrapper script is invoked while on a networked FS. - info: about 30 requests per run - swift-specific fs access (what the wrapper is meant to do) - application fs requests At 100 jobs/s, only the wrapper causes about 10000 requests/s to the fs server. I suspect that what Allan observed with the moving of the output files being slow is a coincidence. I did a run which showed that for jobs towards the start, the operations towards the end of the wrapper execution are slow, while jobs towards the end have the first part of the wrapper process running slower. This is likely due to ramp-up and ramp-down. I wanted to plot that, but BGP is down today, so it will have to wait. The solution is the having things on the node local FS. Ben already added some code to do that. I changed that a bit and also moved the info file to the scratch fs (unless the user requests that the info be on NFS in order to get progressive results for debugging purposes). A scratch directory different than the work directory is used whenever the user specifies dir in sites.xml. Another thing is using provider job status instead of files when using coasters or falkon. With coasters, scratch FS, and provider status, I empirically determined that an average throughput of 100jobs/s is something that the system (swift + coasters) can sustain well, provided that swift tries to keep the number of jobs submitted to the coaster service to about twice the number of workers. I tested this with 6000 workers and 60 second jobs. I will post the plots shortly. So here's how one would go with this on intrepid: - determine the maximum number of workers (avg-exec-time * 100) - set the nodeGranularity to 512 nodes, 4 workers per node. Also set maxWorkers to 512 so that only 512 node blocks are requested. For some reason 512 node partitions start almost instantly (even if you have 6 of them) while 1024 node partitions you have to wait for. - set the total number of blocks ("slots" parameter) to no-of-workers/2048. - set the jobThrottle to 2*no-of-workers/100 - make sure you also have foreach,max.threads set to 2*no-of-workers (though that depends on the structure of the program). - run on login6. There is no point in using the normal login machines since they have a limit of 1024 file descriptors per process. I will actually code an xml element for sites.xml to capture this without that much pain. There is eventually a hard limit of (a bit less than) 65536 workers. I think. This is because each TCP connection from the workers requires a local port on the coaster service side, and there's a limit of 2^16 to that. This could eventually be addressed by having proxies on the IO nodes or something. On intrepid (as opposed to surveyor) the default queue won't accept 1-node jobs. So the cleanup job at the end of a run will fail with a nice display of a stack trace and you will have to manually clean up the work directory. From hategan at mcs.anl.gov Mon Oct 26 12:06:42 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 26 Oct 2009 12:06:42 -0500 Subject: [Swift-devel] Swift and BGP plots Message-ID: <1256576802.4067.3.camel@localhost> 16k jobs, scratch on GPFS: http://www.mcs.anl.gov/~hategan/report-bgp-plain/ 16k jobs, scratch on compute node: http://www.mcs.anl.gov/~hategan/report-bgp-scratch/ 16k jobs, scratch on compute node, status through provider: http://www.mcs.anl.gov/~hategan/report-bgp-scratch-provider/ 64k jobs, 4000 workers: http://www.mcs.anl.gov/~hategan/report-dc-4000/ 64k jobs, 6000 workers: http://www.mcs.anl.gov/~hategan/report-dc-6000/ From foster at anl.gov Mon Oct 26 15:07:34 2009 From: foster at anl.gov (Ian Foster) Date: Mon, 26 Oct 2009 15:07:34 -0500 Subject: [Swift-devel] Swift and BGP In-Reply-To: <1256576209.1135.46.camel@localhost> References: <1256576209.1135.46.camel@localhost> Message-ID: <924B1323-386B-4E2C-8B97-B0C54E6FC1FB@anl.gov> Hi Mihael: This is very encouraging. It will be helpful to understand how these numbers compare to Falkon, as Falkon is the one other data point we have on what can be achieved on BGP. Ian. On Oct 26, 2009, at 11:56 AM, Mihael Hategan wrote: > I've been playing with Swift on the BGP the past few days. > > My observation is that with the current swift and reasonable mapping > (such as the one done by the concurrent mapper) filesystem slowness > does > not seem to be caused by contentious access to the same file/ > directory. > Instead, it's the sheer amount of network filesystem requests which > come > from a few sources: > > - bash: every time bash forks a process it closes the current script, > forks the process; after the process is done, bash re-opens the script > file, seeks to the position it left it at and reads the next command. > And typical scripts involve forking a few processes. Our wrapper > script > is invoked while on a networked FS. > - info: about 30 requests per run > - swift-specific fs access (what the wrapper is meant to do) > - application fs requests > > At 100 jobs/s, only the wrapper causes about 10000 requests/s to the > fs > server. > > I suspect that what Allan observed with the moving of the output files > being slow is a coincidence. I did a run which showed that for jobs > towards the start, the operations towards the end of the wrapper > execution are slow, while jobs towards the end have the first part of > the wrapper process running slower. This is likely due to ramp-up and > ramp-down. I wanted to plot that, but BGP is down today, so it will > have > to wait. > > The solution is the having things on the node local FS. Ben already > added some code to do that. I changed that a bit and also moved the > info > file to the scratch fs (unless the user requests that the info be on > NFS > in order to get progressive results for debugging purposes). A scratch > directory different than the work directory is used whenever the user > specifies dir in sites.xml. > > Another thing is using provider job status instead of files when using > coasters or falkon. > > With coasters, scratch FS, and provider status, I empirically > determined > that an average throughput of 100jobs/s is something that the system > (swift + coasters) can sustain well, provided that swift tries to keep > the number of jobs submitted to the coaster service to about twice the > number of workers. I tested this with 6000 workers and 60 second > jobs. I > will post the plots shortly. > > So here's how one would go with this on intrepid: > - determine the maximum number of workers (avg-exec-time * 100) > - set the nodeGranularity to 512 nodes, 4 workers per node. Also set > maxWorkers to 512 so that only 512 node blocks are requested. For some > reason 512 node partitions start almost instantly (even if you have > 6 of > them) while 1024 node partitions you have to wait for. > - set the total number of blocks ("slots" parameter) to > no-of-workers/2048. > - set the jobThrottle to 2*no-of-workers/100 > - make sure you also have foreach,max.threads set to 2*no-of-workers > (though that depends on the structure of the program). > - run on login6. There is no point in using the normal login machines > since they have a limit of 1024 file descriptors per process. > > I will actually code an xml element for sites.xml to capture this > without that much pain. > > There is eventually a hard limit of (a bit less than) 65536 workers. I > think. This is because each TCP connection from the workers requires a > local port on the coaster service side, and there's a limit of 2^16 to > that. This could eventually be addressed by having proxies on the IO > nodes or something. > > On intrepid (as opposed to surveyor) the default queue won't accept > 1-node jobs. So the cleanup job at the end of a run will fail with a > nice display of a stack trace and you will have to manually clean up > the > work directory. > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel From hategan at mcs.anl.gov Mon Oct 26 15:36:26 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 26 Oct 2009 15:36:26 -0500 Subject: [Swift-devel] Swift and BGP In-Reply-To: <924B1323-386B-4E2C-8B97-B0C54E6FC1FB@anl.gov> References: <1256576209.1135.46.camel@localhost> <924B1323-386B-4E2C-8B97-B0C54E6FC1FB@anl.gov> Message-ID: <1256589386.7792.23.camel@localhost> On Mon, 2009-10-26 at 15:07 -0500, Ian Foster wrote: > Hi Mihael: > > This is very encouraging. > > It will be helpful to understand how these numbers compare to Falkon, > as Falkon is the one other data point we have on what can be achieved > on BGP. Sure. Though I suspect swift limits are reached before falkon or coaster limits are reached. So unless we compare raw falkon and raw coasters, we probably won't see much of a difference. Which would then be pretty useless for Swift. From hategan at mcs.anl.gov Mon Oct 26 15:48:47 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 26 Oct 2009 15:48:47 -0500 Subject: [Swift-devel] Swift and BGP In-Reply-To: <1256576209.1135.46.camel@localhost> References: <1256576209.1135.46.camel@localhost> Message-ID: <1256590127.9206.6.camel@localhost> On Mon, 2009-10-26 at 11:56 -0500, Mihael Hategan wrote: > So here's how one would go with this on intrepid: > - determine the maximum number of workers (avg-exec-time * 100) > - set the nodeGranularity to 512 nodes, 4 workers per node. Also set > maxWorkers to 512 so that only 512 node blocks are requested. For some > reason 512 node partitions start almost instantly (even if you have 6 of > them) while 1024 node partitions you have to wait for. > - set the total number of blocks ("slots" parameter) to > no-of-workers/2048. > - set the jobThrottle to 2*no-of-workers/100 > - make sure you also have foreach,max.threads set to 2*no-of-workers > (though that depends on the structure of the program). > - run on login6. There is no point in using the normal login machines > since they have a limit of 1024 file descriptors per process. > > I will actually code an xml element for sites.xml to capture this > without that much pain. In sites.xml under the intrepid pool element, you can use: This sets relevant stuff if averageJobTime is specified. Default block size is 512 nodes. It will print the settings. From iraicu at cs.uchicago.edu Mon Oct 26 16:36:00 2009 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Mon, 26 Oct 2009 16:36:00 -0500 Subject: [Swift-devel] Swift and BGP plots In-Reply-To: <1256576802.4067.3.camel@localhost> References: <1256576802.4067.3.camel@localhost> Message-ID: <4AE61640.9080001@cs.uchicago.edu> Hi Mihael, This is interesting stuff! Here is what I understood from the following figures (given the summary of the execute2 tab): Mihael Hategan wrote: > 16k jobs, scratch on GPFS: > http://www.mcs.anl.gov/~hategan/report-bgp-plain/ > Shortest event (s): 17.1009998321533 Longest event (s): 521.828999996185 Mean event duration (s): 358.316546020797 Standard deviation of event duration (s): 109.087010337575 > 16k jobs, scratch on compute node: > http://www.mcs.anl.gov/~hategan/report-bgp-scratch/ > Shortest event (s): 11.606999874115 Longest event (s): 376.588999986649 Mean event duration (s): 176.960421203068 Standard deviation of event duration (s): 75.0991380521202 > 16k jobs, scratch on compute node, status through provider: > http://www.mcs.anl.gov/~hategan/report-bgp-scratch-provider/ > Shortest event (s): 11.8900001049042 Longest event (s): 223.809000015259 Mean event duration (s): 135.653097596136 Standard deviation of event duration (s): 62.1117594571245 For all the above stats, I don't understand why the minimum time is 10~20 seconds, when the jobs are sleep 60? Or perhaps you were doing sleep 0 here? > 64k jobs, 4000 workers: > http://www.mcs.anl.gov/~hategan/report-dc-4000/ > Shortest event (s): 106.119999885559 Longest event (s): 1246.60699987411 Mean event duration (s): 334.987874176266 Standard deviation of event duration (s): 290.212811366649 Efficiency: 18%? > 64k jobs, 6000 workers: > http://www.mcs.anl.gov/~hategan/report-dc-6000/ > Shortest event (s): 108.671000003815 Longest event (s): 940.963999986649 Mean event duration (s): 255.069875579873 Standard deviation of event duration (s): 130.231747714145 Efficiency: 23.5%? I assume this is all with "sleep 60" jobs, right? Here are some comparisons of raw Falkon: 20K jobs, 2K workers, sleep 32, single Falkon service running on login6 Mean event duration (s): 32.6798 Efficiency: 98.3% 40K jobs, 4K workers, sleep 32, distributed Falkon service running on 16 I/O nodes Mean event duration (s): 34.659 Efficiency: 92.3% 40K jobs, 4K workers, sleep 64, distributed Falkon service running on 16 I/O nodes Mean event duration (s): 67.02667 Efficiency: 95.5% 1M jobs (983,040), 160K workers, sleep 64, distributed Falkon service running on 640 I/O nodes Mean event duration (s): 70.71823 Efficiency: 90.5% I did a search through my inbox for an old email from Zhao Zhang. Here is the summary of a run he made: Total number of events: 512 Shortest event (s): 30 Longest event (s): 33 Mean event duration (s): 30.986328125 Standard deviation of event duration (s): 0.784249612581342 Efficiency: 96.8% I believe in this run, he had 512 jobs of sleep 30, running on 256 workers via Falkon, using Swift. This small scale run, got a 96.8% efficiency, which seemed great! He used to have some logs online at http://www.ci.uchicago.edu/~zzhang/report-sleep-20081016-1808-96bsfgec/, but they are not there anymore. Perhaps Zhao still has these plots. I believe he might have been using some of his CIO (collective I/O) optimizations in these runs. I can't seem to find larger scale runs with these optimizations. These numbers that I posted above, are bits and pieces I found through various experiments I found we ran over the last year, so their direct comparisons is not apples to apples, as things have evolved, Swift, Falkon, including the file system GPFS on the BG/P. I certainly think it would be useful to have a detailed comparison of Swift+Coaster and Swift+Falkon using the latest Swift. Cheers, Ioan > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= From iraicu at cs.uchicago.edu Mon Oct 26 16:36:15 2009 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Mon, 26 Oct 2009 16:36:15 -0500 Subject: [Swift-devel] Swift and BGP In-Reply-To: <1256576209.1135.46.camel@localhost> References: <1256576209.1135.46.camel@localhost> Message-ID: <4AE6164F.2050808@cs.uchicago.edu> Mihael Hategan wrote: > I've been playing with Swift on the BGP the past few days. > > My observation is that with the current swift and reasonable mapping > (such as the one done by the concurrent mapper) filesystem slowness does > not seem to be caused by contentious access to the same file/directory. > Instead, it's the sheer amount of network filesystem requests which come > from a few sources: > > - bash: every time bash forks a process it closes the current script, > forks the process; after the process is done, bash re-opens the script > file, seeks to the position it left it at and reads the next command. > And typical scripts involve forking a few processes. Our wrapper script > is invoked while on a networked FS. > - info: about 30 requests per run > - swift-specific fs access (what the wrapper is meant to do) > - application fs requests > > At 100 jobs/s, only the wrapper causes about 10000 requests/s to the fs > server. > Here were our experiences with running scripts from GPFS. The #s below represents the throughput for invoking scripts (a bash script that invoked a sleep 0) from GPFS on 4 workers, 256 workers, and 2048 workers. Number of Processors Invoke script throughput (ops/sec) 4 125.214 256 109.3272 2048 823.0374 > I suspect that what Allan observed with the moving of the output files > being slow is a coincidence. I did a run which showed that for jobs > towards the start, the operations towards the end of the wrapper > execution are slow, while jobs towards the end have the first part of > the wrapper process running slower. This is likely due to ramp-up and > ramp-down. I wanted to plot that, but BGP is down today, so it will have > to wait. > > The solution is the having things on the node local FS. Ben already > added some code to do that. I changed that a bit and also moved the info > file to the scratch fs (unless the user requests that the info be on NFS > in order to get progressive results for debugging purposes). A scratch > directory different than the work directory is used whenever the user > specifies dir in sites.xml. > > Another thing is using provider job status instead of files when using > coasters or falkon. > > With coasters, scratch FS, and provider status, I empirically determined > that an average throughput of 100jobs/s is something that the system > (swift + coasters) can sustain well, provided that swift tries to keep > the number of jobs submitted to the coaster service to about twice the > number of workers. I tested this with 6000 workers and 60 second jobs. I > will post the plots shortly. > > So here's how one would go with this on intrepid: > - determine the maximum number of workers (avg-exec-time * 100) > - set the nodeGranularity to 512 nodes, 4 workers per node. Also set > maxWorkers to 512 so that only 512 node blocks are requested. For some > reason 512 node partitions start almost instantly (even if you have 6 of > them) while 1024 node partitions you have to wait for. > - set the total number of blocks ("slots" parameter) to > no-of-workers/2048. > - set the jobThrottle to 2*no-of-workers/100 > - make sure you also have foreach,max.threads set to 2*no-of-workers > (though that depends on the structure of the program). > - run on login6. There is no point in using the normal login machines > since they have a limit of 1024 file descriptors per process. > > I will actually code an xml element for sites.xml to capture this > without that much pain. > > There is eventually a hard limit of (a bit less than) 65536 workers. I > think. This is because each TCP connection from the workers requires a > local port on the coaster service side, and there's a limit of 2^16 to > that. This could eventually be addressed by having proxies on the IO > nodes or something. > In our experience with Falkon, the limit came much sooner than 64K. In Falkon, using the C worker code (which runs on the BG/P), each worker consumes 2 TCP/IP connections to the Falkon service. In the centralized Falkon service version, this racks up connections pretty quick. I don't recall at exactly what point we started having issues, but it was somewhere in the range of 10K~20K CPU cores. Essentially, we could establish all the connections (20K~40K TCP connections), but when the experiment would actually start, and data needed to flow over these connections, all sort of weird stuff started happening, TCP connection would get reset, workers were failing (e.g. their TCP connection was being severed and not being re-established), etc. I want to say that 8K (maybe 16K) cores was the largest tests we made on the BG/P with a centralized Falkon service, that were stable and successful. However, even at these scales, the sys admins were complaining that we were running the login nodes too hard, and we couldn't scale much more, without taking over multiple login nodes ;) So, we were motivated to go the distributed service route, where we ran the Falkon service on the I/O nodes, and each service only managed 256 cores. This approach adds some overhead (especially as the I/O nodes are quite underpowered compared to the login nodes), but it gave us a more scalable solution, that worked great from 256 cores to 160K cores. For the BG/P specifically, I think the distribution of the Falkon service to the I/O nodes gave us a low maintanance, robust, and scalable solution! Ioan > On intrepid (as opposed to surveyor) the default queue won't accept > 1-node jobs. So the cleanup job at the end of a run will fail with a > nice display of a stack trace and you will have to manually clean up the > work directory. > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Mon Oct 26 17:44:46 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 26 Oct 2009 17:44:46 -0500 Subject: [Swift-devel] Swift and BGP plots In-Reply-To: <4AE61640.9080001@cs.uchicago.edu> References: <1256576802.4067.3.camel@localhost> <4AE61640.9080001@cs.uchicago.edu> Message-ID: <1256597086.10196.38.camel@localhost> On Mon, 2009-10-26 at 16:36 -0500, Ioan Raicu wrote: [...] > For all the above stats, I don't understand why the minimum time is > 10~20 seconds, when the jobs are sleep 60? Or perhaps you were doing > sleep 0 here? Well, I don't understand what the log plot tools mean by those numbers, but it's clearly not the job time. > > 64k jobs, 4000 workers: > > http://www.mcs.anl.gov/~hategan/report-dc-4000/ > > > Shortest event (s): 106.119999885559 > Longest event (s): 1246.60699987411 > Mean event duration (s): 334.987874176266 > Standard deviation of event duration (s): 290.212811366649 > > Efficiency: 18%? Well, I was a bit afraid that this will turn out into a numerology exercise. I'm not sure whether to be happy or sad that I was right. I'm not sure what "efficiency" is supposed to mean, but you can look at a few things: 1. The coaster panel. That gives you average utilization of workers in each block measured by the code that sends the requests to the workers. It's obtained by dividing the time a CPU is known to be running a job divided by the time a CPU is known to be running a job plus the time a CPU is known to be sitting idle. I need to revise that a bit to account for delays in sending the messages to the workers, but it should be reasonably accurate. Those numbers are above 99% and therefore look suspiciously high. 2. Multiply 60s with the number of jobs (65535), divide by the number of workers (6*1024) and then by the total time since the first job starts to when the last job finishes (or you could choose the middle of the ramp-up to the middle of the ramp-down to get some sort of amortized efficiency). That gives you about 91% end-to end and 96% amortized. Or you could divide by the total time, including swift startup, partition boot time, etc. to get 64%. >From these you can derive various speedups, by multiplying those percentages with the number of workers (6*1024). [...] From hategan at mcs.anl.gov Mon Oct 26 17:55:30 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 26 Oct 2009 17:55:30 -0500 Subject: [Swift-devel] Swift and BGP In-Reply-To: <4AE6164F.2050808@cs.uchicago.edu> References: <1256576209.1135.46.camel@localhost> <4AE6164F.2050808@cs.uchicago.edu> Message-ID: <1256597730.10196.49.camel@localhost> On Mon, 2009-10-26 at 16:36 -0500, Ioan Raicu wrote: > > > Here were our experiences with running scripts from GPFS. The #s below > represents the throughput for invoking scripts (a bash script that > invoked a sleep 0) from GPFS on 4 workers, 256 workers, and 2048 > workers. > Number of Processors > Invoke script throughput (ops/sec) > 4 > 125.214 > 256 > 109.3272 > 2048 > 823.0374 > Looks right. What I saw was that things were getting shitty at around 10000 cores. Lower if info writing, directory making, and file copying was involved. > > [...] > In our experience with Falkon, the limit came much sooner than 64K. In > Falkon, using the C worker code (which runs on the BG/P), each worker > consumes 2 TCP/IP connections to the Falkon service. Well, the coaster workers use only one connection. > In the centralized Falkon service version, this racks up connections > pretty quick. I don't recall at exactly what point we started having > issues, but it was somewhere in the range of 10K~20K CPU cores. > Essentially, we could establish all the connections (20K~40K TCP > connections), but when the experiment would actually start, and data > needed to flow over these connections, all sort of weird stuff started > happening, TCP connection would get reset, workers were failing (e.g. > their TCP connection was being severed and not being re-established), > etc. I want to say that 8K (maybe 16K) cores was the largest tests we > made on the BG/P with a centralized Falkon service, that were stable > and successful. Possible. I haven't properly tested above 12k workers. I was just mentioning a theoretical limitation that doesn't seem possible to beat without having things distributed. [...] > > For the BG/P specifically, I think the distribution of the Falkon > service to the I/O nodes gave us a low maintanance, robust, and > scalable solution! Lower than if you only had to run one service on the head node? From iraicu at cs.uchicago.edu Mon Oct 26 22:16:28 2009 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Mon, 26 Oct 2009 22:16:28 -0500 Subject: [Fwd: Re: [Swift-devel] Swift and BGP] Message-ID: <4AE6660C.8020309@cs.uchicago.edu> Resending from my UChicago account, as my previous email from Northwestern bounced. Ioan -------- Original Message -------- Subject: Re: [Swift-devel] Swift and BGP Date: Mon, 26 Oct 2009 16:51:56 -0500 From: Ioan Raicu Reply-To: iraicu at eecs.northwestern.edu Organization: Northwestern University, Department of Electrical Engineering and Computer Science To: Mihael Hategan CC: Ian Foster , swift-devel References: <1256576209.1135.46.camel at localhost> <924B1323-386B-4E2C-8B97-B0C54E6FC1FB at anl.gov> <1256589386.7792.23.camel at localhost> Although, I don't know if we understand the Swift limits all that well. We never managed to do a systematic performance evaluation of Swift (with the exception of the initial 2007 results from the SWF07 paper), or at least I am not aware of it. Perhaps this is a good excuse to do such a performance evaluation. Ioan -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= Mihael Hategan wrote: > On Mon, 2009-10-26 at 15:07 -0500, Ian Foster wrote: > >> Hi Mihael: >> >> This is very encouraging. >> >> It will be helpful to understand how these numbers compare to Falkon, >> as Falkon is the one other data point we have on what can be achieved >> on BGP. >> > > Sure. Though I suspect swift limits are reached before falkon or coaster > limits are reached. So unless we compare raw falkon and raw coasters, we > probably won't see much of a difference. Which would then be pretty > useless for Swift. > > -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Mon Oct 26 22:54:31 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 26 Oct 2009 22:54:31 -0500 Subject: [Fwd: Re: [Swift-devel] Swift and BGP] In-Reply-To: <4AE6660C.8020309@cs.uchicago.edu> References: <4AE6660C.8020309@cs.uchicago.edu> Message-ID: <1256615671.15438.36.camel@localhost> On Mon, 2009-10-26 at 22:16 -0500, Ioan Raicu wrote: > Although, I don't know if we understand the Swift limits all that > well. We never managed to do a systematic performance evaluation of > Swift (with the exception of the initial 2007 results from the SWF07 > paper), or at least I am not aware of it. Perhaps this is a good > excuse to do such a performance evaluation. Yep. From iraicu at cs.uchicago.edu Mon Oct 26 23:19:01 2009 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Mon, 26 Oct 2009 23:19:01 -0500 Subject: [Swift-devel] Swift and BGP plots In-Reply-To: <1256597086.10196.38.camel@localhost> References: <1256576802.4067.3.camel@localhost> <4AE61640.9080001@cs.uchicago.edu> <1256597086.10196.38.camel@localhost> Message-ID: <4AE674B5.5000406@cs.uchicago.edu> Mihael Hategan wrote: > On Mon, 2009-10-26 at 16:36 -0500, Ioan Raicu wrote: > [...] > > >> For all the above stats, I don't understand why the minimum time is >> 10~20 seconds, when the jobs are sleep 60? Or perhaps you were doing >> sleep 0 here? >> > > Well, I don't understand what the log plot tools mean by those numbers, > but it's clearly not the job time. > OK. So which plot/data should I be looking at to get the summary of the per tasks performance? > >>> 64k jobs, 4000 workers: >>> http://www.mcs.anl.gov/~hategan/report-dc-4000/ >>> >>> >> Shortest event (s): 106.119999885559 >> Longest event (s): 1246.60699987411 >> Mean event duration (s): 334.987874176266 >> Standard deviation of event duration (s): 290.212811366649 >> >> Efficiency: 18%? >> > > Well, I was a bit afraid that this will turn out into a numerology > exercise. I'm not sure whether to be happy or sad that I was right. > > I'm not sure what "efficiency" is supposed to mean, The simplest definition of efficiency I used here, was: ideal event duration / mean event duration In the case of the above 18%, I took 60 / 334 ~ 0.18 = 18% This doesn't take into account the ramp up, and ramp down time. In reality, the actual end-to-end efficiency is usually even lower. > but you can look at > a few things: > 1. The coaster panel. That gives you average utilization of workers in > each block measured by the code that sends the requests to the workers. > It's obtained by dividing the time a CPU is known to be running a job > divided by the time a CPU is known to be running a job plus the time a > CPU is known to be sitting idle. I need to revise that a bit to account > for delays in sending the messages to the workers, but it should be > reasonably accurate. Those numbers are above 99% and therefore look > suspiciously high. > I see the block utilization near 100% all the time, so that doesn't seem to match the other data I saw. > 2. Multiply 60s with the number of jobs (65535), divide by the number of > workers (6*1024) and then by the total time since the first job starts > to when the last job finishes (or you could choose the middle of the > ramp-up to the middle of the ramp-down to get some sort of amortized > efficiency). That gives you about 91% end-to end and 96% amortized. Or > you could divide by the total time, including swift startup, partition > boot time, etc. to get 64%. > 65535*60/(6*1024) ~ 640 sec. I see the end-to-end time being about 1300 sec, or 1100 sec if we look at just Karajan. The 64% efficiency is in the ballpark, but I don't see where the 91% and 96% are coming from. Under Karajan tab (http://www.mcs.anl.gov/~hategan/report-dc-4000/karajan.html), at the end of the page, I found: Total number of events: 131076 Shortest event (s): 0 Longest event (s): 1200.02400016785 Total duration of all events (s): 8119724.183007 Mean event duration (s): 61.9466888141765 Standard deviation of event duration (s): 4.95689953321352 Maximum number of events at one time: 8194 This looks better in a sense, 61.95 sec mean, with 4.95 sec std dev. Although, it still looks a bit odd, as the shortest event is 0 sec, and longest is 1200 sec. This would infer an efficiency of 96%, which matches what you said above. Where are the Swift logs for these runs? I have a tool that will convert Swift logs to Falkon log format, so I might understand better the data. Ioan > >From these you can derive various speedups, by multiplying those > percentages with the number of workers (6*1024). > > [...] > > > > -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From iraicu at cs.uchicago.edu Mon Oct 26 23:27:24 2009 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Mon, 26 Oct 2009 23:27:24 -0500 Subject: [Swift-devel] Swift and BGP In-Reply-To: <1256597730.10196.49.camel@localhost> References: <1256576209.1135.46.camel@localhost> <4AE6164F.2050808@cs.uchicago.edu> <1256597730.10196.49.camel@localhost> Message-ID: <4AE676AC.6020700@cs.uchicago.edu> Mihael Hategan wrote: > On Mon, 2009-10-26 at 16:36 -0500, Ioan Raicu wrote: > >>> >>> >> Here were our experiences with running scripts from GPFS. The #s below >> represents the throughput for invoking scripts (a bash script that >> invoked a sleep 0) from GPFS on 4 workers, 256 workers, and 2048 >> workers. >> Number of Processors >> Invoke script throughput (ops/sec) >> 4 >> 125.214 >> 256 >> 109.3272 >> 2048 >> 823.0374 >> >> > > Looks right. What I saw was that things were getting shitty at around > 10000 cores. Lower if info writing, directory making, and file copying > was involved. > Right. > >>> [...] >>> >> In our experience with Falkon, the limit came much sooner than 64K. In >> Falkon, using the C worker code (which runs on the BG/P), each worker >> consumes 2 TCP/IP connections to the Falkon service. >> > > Well, the coaster workers use only one connection. > Its 1 connection per core? or per node? Zhao tried to reduce to 1 connection per node, but the worker was not stable, so we left it alone in the interest of time. The last time I looked at it, the workers used 2 connections per core, or 8 connections per node. Quite inefficient at scale, but not an issue given that each service only handles 256 cores. > >> In the centralized Falkon service version, this racks up connections >> pretty quick. I don't recall at exactly what point we started having >> issues, but it was somewhere in the range of 10K~20K CPU cores. >> Essentially, we could establish all the connections (20K~40K TCP >> connections), but when the experiment would actually start, and data >> needed to flow over these connections, all sort of weird stuff started >> happening, TCP connection would get reset, workers were failing (e.g. >> their TCP connection was being severed and not being re-established), >> etc. I want to say that 8K (maybe 16K) cores was the largest tests we >> made on the BG/P with a centralized Falkon service, that were stable >> and successful. >> > > Possible. I haven't properly tested above 12k workers. I was just > mentioning a theoretical limitation that doesn't seem possible to beat > without having things distributed. > > [...] > >> For the BG/P specifically, I think the distribution of the Falkon >> service to the I/O nodes gave us a low maintanance, robust, and >> scalable solution! >> > > Lower than if you only had to run one service on the head node? > Yes, in fact it was for Falkon. If we ran Falkon on the head node, the user would have to start it manually, on an available port, and then shut it down when finished. Running things on the I/O nodes was tougher at the beginning, but once we got it all configured and running, it was great! The Falkon service starts up on I/O node boot time, on a specific port (no need to check if its available as the I/O node is dedicated to the user), all compute nodes can easily find their respective I/O nodes at the same location (some 192.xxx private address), and when the run is over, the I/O nodes terminate and the services stop all on their own. At least for Falkon, it really made the difference between having a turn-key solution that always works, and one that would require constant tinkering (starting and stopping) and configuration (e.g. ports). Again, the downside to the distributed one, was the overhead of implementing and testing it, and also the load-balancing that required a bit of fine tunning in Swift to get just right. Ioan > > > -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Tue Oct 27 00:35:47 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 27 Oct 2009 00:35:47 -0500 Subject: [Swift-devel] Swift and BGP plots In-Reply-To: <4AE674B5.5000406@cs.uchicago.edu> References: <1256576802.4067.3.camel@localhost> <4AE61640.9080001@cs.uchicago.edu> <1256597086.10196.38.camel@localhost> <4AE674B5.5000406@cs.uchicago.edu> Message-ID: <1256621747.17056.67.camel@localhost> On Mon, 2009-10-26 at 23:19 -0500, Ioan Raicu wrote: > > > Mihael Hategan wrote: > > On Mon, 2009-10-26 at 16:36 -0500, Ioan Raicu wrote: > > [...] [...] > > > OK. So which plot/data should I be looking at to get the summary of > the per tasks performance? None. You calculate the efficiency based on the total time, individual task time and number of nodes. > > > > 64k jobs, 4000 workers: > > > > http://www.mcs.anl.gov/~hategan/report-dc-4000/ > > > > > > > > > > > Shortest event (s): 106.119999885559 > > > Longest event (s): 1246.60699987411 > > > Mean event duration (s): 334.987874176266 > > > Standard deviation of event duration (s): 290.212811366649 [...] > In the case of the above 18%, I took 60 / 334 ~ 0.18 = 18% See http://en.wikipedia.org/wiki/Speedup Efficiency is speedup divided by number of cores. You cannot infer things from the mean duration because it says nothing about the degree of parallelism. I.e. you're not interested in how fast individual things are going, but how fast all things are going overall. That's why you can have a crappy-cpu machine like the BGP be the in the top 10. Allow me to paint this: Scenario 1: [-1-][-2-][-1-][-2-] Scenario 2: [---1----][---2----] (how two tasks are scheduled by a scheduler on one CPU). Efficiency is the same in both cases. In scenario 1 average task duration is 15 characters, in scenario 2 it's 10 characters. In both cases the raw task duration is 10 characters. [...] > > > I see the block utilization near 100% all the time, Right. That's because it measures the wrapper time, not the wrapped job time (some time is spent doing whatever the wrapper does). That just says that the job-to-worker dispatch algorithm in the coasters works ok with that load. > so that doesn't seem to match the other data I saw. They measure different things. But if you calculate the efficiency the proper way, you'll see that they are closer. > > 2. Multiply 60s with the number of jobs (65535), divide by the number of > > workers (6*1024) and then by the total time since the first job starts > > to when the last job finishes (or you could choose the middle of the > > ramp-up to the middle of the ramp-down to get some sort of amortized > > efficiency). That gives you about 91% end-to end and 96% amortized. Or > > you could divide by the total time, including swift startup, partition > > boot time, etc. to get 64%. > > > 65535*60/(6*1024) ~ 640 sec. I see the end-to-end time being about > 1300 sec, or 1100 sec if we look at just Karajan. The 64% efficiency > is in the ballpark, but I don't see where the 91% and 96% are coming > from. I think you're mixing the runs. Sorry I didn't make it more clear, but dc-4000 is the 4*1024 core run and dc-6000 is the 6*1024 core run. So you're dividing the 4*1024 core speedup by 6*1024 which gives you 2/3 of the efficiency. Multiply 64% by 3/2 to get the proper number back. From hategan at mcs.anl.gov Tue Oct 27 00:39:32 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 27 Oct 2009 00:39:32 -0500 Subject: [Swift-devel] Swift and BGP In-Reply-To: <4AE676AC.6020700@cs.uchicago.edu> References: <1256576209.1135.46.camel@localhost> <4AE6164F.2050808@cs.uchicago.edu> <1256597730.10196.49.camel@localhost> <4AE676AC.6020700@cs.uchicago.edu> Message-ID: <1256621972.17056.71.camel@localhost> On Mon, 2009-10-26 at 23:27 -0500, Ioan Raicu wrote: > > > > > > > In our experience with Falkon, the limit came much sooner than 64K. In > > > Falkon, using the C worker code (which runs on the BG/P), each worker > > > consumes 2 TCP/IP connections to the Falkon service. > > > > > > > Well, the coaster workers use only one connection. > > > Its 1 connection per core? or per node? Per core. > Zhao tried to reduce to 1 connection per node, but the worker was not > stable, so we left it alone in the interest of time. That's an awesome idea! Thanks! From iraicu at cs.uchicago.edu Tue Oct 27 00:41:52 2009 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Tue, 27 Oct 2009 00:41:52 -0500 Subject: [Swift-devel] Swift and BGP plots In-Reply-To: <1256621747.17056.67.camel@localhost> References: <1256576802.4067.3.camel@localhost> <4AE61640.9080001@cs.uchicago.edu> <1256597086.10196.38.camel@localhost> <4AE674B5.5000406@cs.uchicago.edu> <1256621747.17056.67.camel@localhost> Message-ID: <4AE68820.3050606@cs.uchicago.edu> Where can I get the Swift logs for these runs, I'd like to pass them through my Swift-to-Falkon conversion tool :) Ioan Mihael Hategan wrote: > On Mon, 2009-10-26 at 23:19 -0500, Ioan Raicu wrote: > >> Mihael Hategan wrote: >> >>> On Mon, 2009-10-26 at 16:36 -0500, Ioan Raicu wrote: >>> [...] >>> > [...] > >>> >>> >> OK. So which plot/data should I be looking at to get the summary of >> the per tasks performance? >> > > None. You calculate the efficiency based on the total time, individual > task time and number of nodes. > > >>>>> 64k jobs, 4000 workers: >>>>> http://www.mcs.anl.gov/~hategan/report-dc-4000/ >>>>> >>>>> >>>>> >>>> Shortest event (s): 106.119999885559 >>>> Longest event (s): 1246.60699987411 >>>> Mean event duration (s): 334.987874176266 >>>> Standard deviation of event duration (s): 290.212811366649 >>>> > [...] > >> In the case of the above 18%, I took 60 / 334 ~ 0.18 = 18% >> > > See http://en.wikipedia.org/wiki/Speedup > > Efficiency is speedup divided by number of cores. You cannot infer > things from the mean duration because it says nothing about the degree > of parallelism. I.e. you're not interested in how fast individual things > are going, but how fast all things are going overall. That's why you can > have a crappy-cpu machine like the BGP be the in the top 10. Allow me to > paint this: > > Scenario 1: [-1-][-2-][-1-][-2-] > Scenario 2: [---1----][---2----] > > (how two tasks are scheduled by a scheduler on one CPU). > > Efficiency is the same in both cases. In scenario 1 average task > duration is 15 characters, in scenario 2 it's 10 characters. In both > cases the raw task duration is 10 characters. > > [...] > >>> >>> >> I see the block utilization near 100% all the time, >> > > Right. That's because it measures the wrapper time, not the wrapped job > time (some time is spent doing whatever the wrapper does). That just > says that the job-to-worker dispatch algorithm in the coasters works ok > with that load. > > >> so that doesn't seem to match the other data I saw. >> > > They measure different things. But if you calculate the efficiency the > proper way, you'll see that they are closer. > > >>> 2. Multiply 60s with the number of jobs (65535), divide by the number of >>> workers (6*1024) and then by the total time since the first job starts >>> to when the last job finishes (or you could choose the middle of the >>> ramp-up to the middle of the ramp-down to get some sort of amortized >>> efficiency). That gives you about 91% end-to end and 96% amortized. Or >>> you could divide by the total time, including swift startup, partition >>> boot time, etc. to get 64%. >>> >>> >> 65535*60/(6*1024) ~ 640 sec. I see the end-to-end time being about >> 1300 sec, or 1100 sec if we look at just Karajan. The 64% efficiency >> is in the ballpark, but I don't see where the 91% and 96% are coming >> from. >> > > I think you're mixing the runs. Sorry I didn't make it more clear, but > dc-4000 is the 4*1024 core run and dc-6000 is the 6*1024 core run. So > you're dividing the 4*1024 core speedup by 6*1024 which gives you 2/3 of > the efficiency. Multiply 64% by 3/2 to get the proper number back. > > > > -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Tue Oct 27 00:51:09 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 27 Oct 2009 00:51:09 -0500 Subject: [Swift-devel] Swift and BGP plots In-Reply-To: <4AE68820.3050606@cs.uchicago.edu> References: <1256576802.4067.3.camel@localhost> <4AE61640.9080001@cs.uchicago.edu> <1256597086.10196.38.camel@localhost> <4AE674B5.5000406@cs.uchicago.edu> <1256621747.17056.67.camel@localhost> <4AE68820.3050606@cs.uchicago.edu> Message-ID: <1256622669.18682.0.camel@localhost> On Tue, 2009-10-27 at 00:41 -0500, Ioan Raicu wrote: > Where can I get the Swift logs for these runs, I'd like to pass them > through my Swift-to-Falkon conversion tool :) I ignored that because I was going to post them tomorrow. So hang on until then. From hategan at mcs.anl.gov Tue Oct 27 12:45:02 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 27 Oct 2009 12:45:02 -0500 Subject: [Swift-devel] Swift and BGP plots In-Reply-To: <1256622669.18682.0.camel@localhost> References: <1256576802.4067.3.camel@localhost> <4AE61640.9080001@cs.uchicago.edu> <1256597086.10196.38.camel@localhost> <4AE674B5.5000406@cs.uchicago.edu> <1256621747.17056.67.camel@localhost> <4AE68820.3050606@cs.uchicago.edu> <1256622669.18682.0.camel@localhost> Message-ID: <1256665502.23397.0.camel@localhost> On Tue, 2009-10-27 at 00:51 -0500, Mihael Hategan wrote: > On Tue, 2009-10-27 at 00:41 -0500, Ioan Raicu wrote: > > Where can I get the Swift logs for these runs, I'd like to pass them > > through my Swift-to-Falkon conversion tool :) > > I ignored that because I was going to post them tomorrow. So hang on > until then. http://www.mcs.anl.gov/~hategan/dc-4000.log.gz http://www.mcs.anl.gov/~hategan/dc-6000.log.gz From iraicu at cs.uchicago.edu Tue Oct 27 14:17:19 2009 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Tue, 27 Oct 2009 14:17:19 -0500 Subject: [Swift-devel] Swift and BGP plots In-Reply-To: <1256665502.23397.0.camel@localhost> References: <1256576802.4067.3.camel@localhost> <4AE61640.9080001@cs.uchicago.edu> <1256597086.10196.38.camel@localhost> <4AE674B5.5000406@cs.uchicago.edu> <1256621747.17056.67.camel@localhost> <4AE68820.3050606@cs.uchicago.edu> <1256622669.18682.0.camel@localhost> <1256665502.23397.0.camel@localhost> Message-ID: <4AE7473F.4010404@cs.uchicago.edu> OK, I converted the logs, and here is what I got. The thing that bugs me is that things ran faster (per task) at the larger scale (a bit counter intuitive). Also, the maximum number of concurrent tasks I found was 20K, in both experiments. This doesn't seem right, as the experiments should have 4K or 6K active tasks at any time. Could I be looking at the wrong entries from the Swift logs? I did this to get the data I needed from the Swift logs: cat dc-4000.log | grep "JOB_START" > dc-4000-start-end.txt cat dc-4000.log | grep "JOB_END" >> dc-4000-start-end.txt Is this correct? Or should I be looking for other job events? Ioan 4K run: Average Task Execution Time (ms): 326663.0046845912 +/- 283374.34801782423 6K run: Average Task Execution Time (ms): 246982.0736259041 +/- 123059.82258438379 Mihael Hategan wrote: > On Tue, 2009-10-27 at 00:51 -0500, Mihael Hategan wrote: > >> On Tue, 2009-10-27 at 00:41 -0500, Ioan Raicu wrote: >> >>> Where can I get the Swift logs for these runs, I'd like to pass them >>> through my Swift-to-Falkon conversion tool :) >>> >> I ignored that because I was going to post them tomorrow. So hang on >> until then. >> > > http://www.mcs.anl.gov/~hategan/dc-4000.log.gz > http://www.mcs.anl.gov/~hategan/dc-6000.log.gz > > > > -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: moz-screenshot-1.jpg Type: image/jpeg Size: 72663 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: moz-screenshot.jpg Type: image/jpeg Size: 74354 bytes Desc: not available URL: From hategan at mcs.anl.gov Tue Oct 27 14:36:18 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 27 Oct 2009 14:36:18 -0500 Subject: [Swift-devel] Swift and BGP plots In-Reply-To: <4AE7473F.4010404@cs.uchicago.edu> References: <1256576802.4067.3.camel@localhost> <4AE61640.9080001@cs.uchicago.edu> <1256597086.10196.38.camel@localhost> <4AE674B5.5000406@cs.uchicago.edu> <1256621747.17056.67.camel@localhost> <4AE68820.3050606@cs.uchicago.edu> <1256622669.18682.0.camel@localhost> <1256665502.23397.0.camel@localhost> <4AE7473F.4010404@cs.uchicago.edu> Message-ID: <1256672178.25578.14.camel@localhost> On Tue, 2009-10-27 at 14:17 -0500, Ioan Raicu wrote: > OK, I converted the logs, and here is what I got. > > The thing that bugs me is that things ran faster (per task) at the > larger scale (a bit counter intuitive). Nope. Same workload but more processors. > Also, the maximum number of concurrent tasks I found was 20K, in both > experiments. This doesn't seem right, as the experiments should have > 4K or 6K active tasks at any time. Right. You should see around 4k or 6k max active tasks. > Could I be looking at the wrong entries from the Swift logs? > > I did this to get the data I needed from the Swift logs: > cat dc-4000.log | grep "JOB_START" > dc-4000-start-end.txt > cat dc-4000.log | grep "JOB_END" >> dc-4000-start-end.txt > > Is this correct? Or should I be looking for other job events? You should probably be looking for "Task(type=JOB_SUBMISSION...) setting status to Active" and "Task(type=JOB_SUBMISSION...) setting status to Completed". Or something like that. JOB_START and JOB_END are statuses for the Swift execute2 processes. From iraicu at cs.uchicago.edu Tue Oct 27 14:48:44 2009 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Tue, 27 Oct 2009 14:48:44 -0500 Subject: [Swift-devel] Swift and BGP plots In-Reply-To: <1256672178.25578.14.camel@localhost> References: <1256576802.4067.3.camel@localhost> <4AE61640.9080001@cs.uchicago.edu> <1256597086.10196.38.camel@localhost> <4AE674B5.5000406@cs.uchicago.edu> <1256621747.17056.67.camel@localhost> <4AE68820.3050606@cs.uchicago.edu> <1256622669.18682.0.camel@localhost> <1256665502.23397.0.camel@localhost> <4AE7473F.4010404@cs.uchicago.edu> <1256672178.25578.14.camel@localhost> Message-ID: <4AE74E9C.5000604@cs.uchicago.edu> OK, so this looks to be a bit more complex then. Can you suggest a way to extract the Active and Complete timestamps (along with the task ID)? It seems that if I grep for JOB_SUBMISSION, the status Active is not found on the same line, so I can't filter according to those two keywords. Also, the Completed keyword also comes up with many other hits. Any ideas on how to get the info I need (with cat, grep, etc) without writing some custom script/program that actually understands in depth the logging of the Swift log? Thanks, Ioan Mihael Hategan wrote: > On Tue, 2009-10-27 at 14:17 -0500, Ioan Raicu wrote: > >> OK, I converted the logs, and here is what I got. >> >> The thing that bugs me is that things ran faster (per task) at the >> larger scale (a bit counter intuitive). >> > > Nope. Same workload but more processors. > > >> Also, the maximum number of concurrent tasks I found was 20K, in both >> experiments. This doesn't seem right, as the experiments should have >> 4K or 6K active tasks at any time. >> > > Right. You should see around 4k or 6k max active tasks. > > >> Could I be looking at the wrong entries from the Swift logs? >> >> I did this to get the data I needed from the Swift logs: >> cat dc-4000.log | grep "JOB_START" > dc-4000-start-end.txt >> cat dc-4000.log | grep "JOB_END" >> dc-4000-start-end.txt >> >> Is this correct? Or should I be looking for other job events? >> > > You should probably be looking for "Task(type=JOB_SUBMISSION...) setting > status to Active" and "Task(type=JOB_SUBMISSION...) setting status to > Completed". Or something like that. > > JOB_START and JOB_END are statuses for the Swift execute2 processes. > > > -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From iraicu at cs.uchicago.edu Tue Oct 27 14:51:51 2009 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Tue, 27 Oct 2009 14:51:51 -0500 Subject: [Swift-devel] Swift and BGP plots In-Reply-To: <4AE74E9C.5000604@cs.uchicago.edu> References: <1256576802.4067.3.camel@localhost> <4AE61640.9080001@cs.uchicago.edu> <1256597086.10196.38.camel@localhost> <4AE674B5.5000406@cs.uchicago.edu> <1256621747.17056.67.camel@localhost> <4AE68820.3050606@cs.uchicago.edu> <1256622669.18682.0.camel@localhost> <1256665502.23397.0.camel@localhost> <4AE7473F.4010404@cs.uchicago.edu> <1256672178.25578.14.camel@localhost> <4AE74E9C.5000604@cs.uchicago.edu> Message-ID: <4AE74F57.5030306@cs.uchicago.edu> How about this: iraicu at diablo:~/falkon/logs/swift/dc-4000> cat dc-4000.log | grep "JOB_SUBMISSION" | grep "Active" iraicu at diablo:~/falkon/logs/swift/dc-4000> cat dc-4000.log | grep "JOB_SUBMISSION" | grep "Completed" This seems to look OK to me, is this what I am looking for? Ioan Ioan Raicu wrote: > OK, so this looks to be a bit more complex then. Can you suggest a way > to extract the Active and Complete timestamps (along with the task > ID)? It seems that if I grep for JOB_SUBMISSION, the status Active is > not found on the same line, so I can't filter according to those two > keywords. Also, the Completed keyword also comes up with many other > hits. Any ideas on how to get the info I need (with cat, grep, etc) > without writing some custom script/program that actually understands > in depth the logging of the Swift log? > > Thanks, > Ioan > > Mihael Hategan wrote: >> On Tue, 2009-10-27 at 14:17 -0500, Ioan Raicu wrote: >> >>> OK, I converted the logs, and here is what I got. >>> >>> The thing that bugs me is that things ran faster (per task) at the >>> larger scale (a bit counter intuitive). >>> >> >> Nope. Same workload but more processors. >> >> >>> Also, the maximum number of concurrent tasks I found was 20K, in both >>> experiments. This doesn't seem right, as the experiments should have >>> 4K or 6K active tasks at any time. >>> >> >> Right. You should see around 4k or 6k max active tasks. >> >> >>> Could I be looking at the wrong entries from the Swift logs? >>> >>> I did this to get the data I needed from the Swift logs: >>> cat dc-4000.log | grep "JOB_START" > dc-4000-start-end.txt >>> cat dc-4000.log | grep "JOB_END" >> dc-4000-start-end.txt >>> >>> Is this correct? Or should I be looking for other job events? >>> >> >> You should probably be looking for "Task(type=JOB_SUBMISSION...) setting >> status to Active" and "Task(type=JOB_SUBMISSION...) setting status to >> Completed". Or something like that. >> >> JOB_START and JOB_END are statuses for the Swift execute2 processes. >> >> >> > > -- > ================================================================= > Ioan Raicu, Ph.D. > NSF/CRA Computing Innovation Fellow > ================================================================= > Center for Ultra-scale Computing and Information Security (CUCIS) > Department of Electrical Engineering and Computer Science > Northwestern University > 2145 Sheridan Rd, Tech M384 > Evanston, IL 60208-3118 > ================================================================= > Cel: 1-847-722-0876 > Tel: 1-847-491-8163 > Email: iraicu at eecs.northwestern.edu > Web: http://www.eecs.northwestern.edu/~iraicu/ > https://wiki.cucis.eecs.northwestern.edu/ > ================================================================= > ================================================================= > > > ------------------------------------------------------------------------ > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Tue Oct 27 14:57:13 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 27 Oct 2009 14:57:13 -0500 Subject: [Swift-devel] Swift and BGP plots In-Reply-To: <4AE74F57.5030306@cs.uchicago.edu> References: <1256576802.4067.3.camel@localhost> <4AE61640.9080001@cs.uchicago.edu> <1256597086.10196.38.camel@localhost> <4AE674B5.5000406@cs.uchicago.edu> <1256621747.17056.67.camel@localhost> <4AE68820.3050606@cs.uchicago.edu> <1256622669.18682.0.camel@localhost> <1256665502.23397.0.camel@localhost> <4AE7473F.4010404@cs.uchicago.edu> <1256672178.25578.14.camel@localhost> <4AE74E9C.5000604@cs.uchicago.edu> <4AE74F57.5030306@cs.uchicago.edu> Message-ID: <1256673433.26427.0.camel@localhost> On Tue, 2009-10-27 at 14:51 -0500, Ioan Raicu wrote: > How about this: > > iraicu at diablo:~/falkon/logs/swift/dc-4000> cat dc-4000.log | grep > "JOB_SUBMISSION" | grep "Active" > iraicu at diablo:~/falkon/logs/swift/dc-4000> cat dc-4000.log | grep > "JOB_SUBMISSION" | grep "Completed" > > This seems to look OK to me, is this what I am looking for? Pretty much grep "JOB_SUBMISSION.*Active" should also work. From iraicu at cs.uchicago.edu Tue Oct 27 15:10:44 2009 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Tue, 27 Oct 2009 15:10:44 -0500 Subject: [Swift-devel] Swift and BGP plots In-Reply-To: <1256673433.26427.0.camel@localhost> References: <1256576802.4067.3.camel@localhost> <4AE61640.9080001@cs.uchicago.edu> <1256597086.10196.38.camel@localhost> <4AE674B5.5000406@cs.uchicago.edu> <1256621747.17056.67.camel@localhost> <4AE68820.3050606@cs.uchicago.edu> <1256622669.18682.0.camel@localhost> <1256665502.23397.0.camel@localhost> <4AE7473F.4010404@cs.uchicago.edu> <1256672178.25578.14.camel@localhost> <4AE74E9C.5000604@cs.uchicago.edu> <4AE74F57.5030306@cs.uchicago.edu> <1256673433.26427.0.camel@localhost> Message-ID: <4AE753C4.9070405@cs.uchicago.edu> Which of these should I use in finding the Complete event? Or are they equivalent? 2009-10-25 21:42:10,717-0500 INFO LateBindingScheduler Task(type=JOB_SUBMISSION, identity=urn:0-1-12173-1-1-1256524749960) Completed. Waiting: 3606, Running: 16387. Heap size: 1003M, Heap free: 66M, Max heap: 1024M 2009-10-25 21:42:10,608-0500 DEBUG TaskImpl Task(type=JOB_SUBMISSION, identity=urn:1256524752479-1256524793584-1256524793585) setting status to Completed Mihael Hategan wrote: > On Tue, 2009-10-27 at 14:51 -0500, Ioan Raicu wrote: > >> How about this: >> >> iraicu at diablo:~/falkon/logs/swift/dc-4000> cat dc-4000.log | grep >> "JOB_SUBMISSION" | grep "Active" >> iraicu at diablo:~/falkon/logs/swift/dc-4000> cat dc-4000.log | grep >> "JOB_SUBMISSION" | grep "Completed" >> >> This seems to look OK to me, is this what I am looking for? >> > > Pretty much > > grep "JOB_SUBMISSION.*Active" should also work. > > > > -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Tue Oct 27 15:16:31 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Tue, 27 Oct 2009 15:16:31 -0500 Subject: [Swift-devel] Swift and BGP plots In-Reply-To: <4AE753C4.9070405@cs.uchicago.edu> References: <1256576802.4067.3.camel@localhost> <4AE61640.9080001@cs.uchicago.edu> <1256597086.10196.38.camel@localhost> <4AE674B5.5000406@cs.uchicago.edu> <1256621747.17056.67.camel@localhost> <4AE68820.3050606@cs.uchicago.edu> <1256622669.18682.0.camel@localhost> <1256665502.23397.0.camel@localhost> <4AE7473F.4010404@cs.uchicago.edu> <1256672178.25578.14.camel@localhost> <4AE74E9C.5000604@cs.uchicago.edu> <4AE74F57.5030306@cs.uchicago.edu> <1256673433.26427.0.camel@localhost> <4AE753C4.9070405@cs.uchicago.edu> Message-ID: <1256674591.26911.1.camel@localhost> On Tue, 2009-10-27 at 15:10 -0500, Ioan Raicu wrote: > Which of these should I use in finding the Complete event? Or are they > equivalent? > > 2009-10-25 21:42:10,717-0500 INFO LateBindingScheduler > Task(type=JOB_SUBMISSION, identity=urn:0-1-12173-1-1-1256524749960) > Completed. Waiting: 3606, Running: 16387. Heap size: 1003M, Heap free: > 66M, Max heap: 1024M > > 2009-10-25 21:42:10,608-0500 DEBUG TaskImpl Task(type=JOB_SUBMISSION, > identity=urn:1256524752479-1256524793584-1256524793585) setting status > to Completed It's probably best if you used TaskImpl for both Active and Completed. From iraicu at cs.uchicago.edu Tue Oct 27 15:24:40 2009 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Tue, 27 Oct 2009 15:24:40 -0500 Subject: [Swift-devel] Swift and BGP plots In-Reply-To: <1256674591.26911.1.camel@localhost> References: <1256576802.4067.3.camel@localhost> <4AE61640.9080001@cs.uchicago.edu> <1256597086.10196.38.camel@localhost> <4AE674B5.5000406@cs.uchicago.edu> <1256621747.17056.67.camel@localhost> <4AE68820.3050606@cs.uchicago.edu> <1256622669.18682.0.camel@localhost> <1256665502.23397.0.camel@localhost> <4AE7473F.4010404@cs.uchicago.edu> <1256672178.25578.14.camel@localhost> <4AE74E9C.5000604@cs.uchicago.edu> <4AE74F57.5030306@cs.uchicago.edu> <1256673433.26427.0.camel@localhost> <4AE753C4.9070405@cs.uchicago.edu> <1256674591.26911.1.camel@localhost> Message-ID: <4AE75708.8020404@cs.uchicago.edu> Great! Mihael Hategan wrote: > On Tue, 2009-10-27 at 15:10 -0500, Ioan Raicu wrote: > >> Which of these should I use in finding the Complete event? Or are they >> equivalent? >> >> 2009-10-25 21:42:10,717-0500 INFO LateBindingScheduler >> Task(type=JOB_SUBMISSION, identity=urn:0-1-12173-1-1-1256524749960) >> Completed. Waiting: 3606, Running: 16387. Heap size: 1003M, Heap free: >> 66M, Max heap: 1024M >> >> 2009-10-25 21:42:10,608-0500 DEBUG TaskImpl Task(type=JOB_SUBMISSION, >> identity=urn:1256524752479-1256524793584-1256524793585) setting status >> to Completed >> > > It's probably best if you used TaskImpl for both Active and Completed. > > > -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From iraicu at cs.uchicago.edu Tue Oct 27 16:03:20 2009 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Tue, 27 Oct 2009 16:03:20 -0500 Subject: [Swift-devel] Swift and BGP plots In-Reply-To: <4AE75708.8020404@cs.uchicago.edu> References: <1256576802.4067.3.camel@localhost> <4AE61640.9080001@cs.uchicago.edu> <1256597086.10196.38.camel@localhost> <4AE674B5.5000406@cs.uchicago.edu> <1256621747.17056.67.camel@localhost> <4AE68820.3050606@cs.uchicago.edu> <1256622669.18682.0.camel@localhost> <1256665502.23397.0.camel@localhost> <4AE7473F.4010404@cs.uchicago.edu> <1256672178.25578.14.camel@localhost> <4AE74E9C.5000604@cs.uchicago.edu> <4AE74F57.5030306@cs.uchicago.edu> <1256673433.26427.0.camel@localhost> <4AE753C4.9070405@cs.uchicago.edu> <1256674591.26911.1.camel@localhost> <4AE75708.8020404@cs.uchicago.edu> Message-ID: <4AE76018.5090007@cs.uchicago.edu> Alright, here it is again: 4K run: Average Task Execution Time (ms): 61946.37017455522 +/- 4884.571426229936 Efficiency: 96.8% 6K run: Average Task Execution Time (ms): 64415.056860142206 +/- 6090.13371098675 Efficiency: 93.1% These both look pretty nice. Now, the only thing that still looks odd, is that in the 4K run, I see 8K active tasks, and in the 6K run I see 12K active tasks. I also have 128K tasks in total. Are my plots off by 2X somehow? Ioan 4K run: 6K run: Ioan Raicu wrote: > Great! > > Mihael Hategan wrote: >> On Tue, 2009-10-27 at 15:10 -0500, Ioan Raicu wrote: >> >>> Which of these should I use in finding the Complete event? Or are they >>> equivalent? >>> >>> 2009-10-25 21:42:10,717-0500 INFO LateBindingScheduler >>> Task(type=JOB_SUBMISSION, identity=urn:0-1-12173-1-1-1256524749960) >>> Completed. Waiting: 3606, Running: 16387. Heap size: 1003M, Heap free: >>> 66M, Max heap: 1024M >>> >>> 2009-10-25 21:42:10,608-0500 DEBUG TaskImpl Task(type=JOB_SUBMISSION, >>> identity=urn:1256524752479-1256524793584-1256524793585) setting status >>> to Completed >>> >> >> It's probably best if you used TaskImpl for both Active and Completed. >> >> >> > > -- > ================================================================= > Ioan Raicu, Ph.D. > NSF/CRA Computing Innovation Fellow > ================================================================= > Center for Ultra-scale Computing and Information Security (CUCIS) > Department of Electrical Engineering and Computer Science > Northwestern University > 2145 Sheridan Rd, Tech M384 > Evanston, IL 60208-3118 > ================================================================= > Cel: 1-847-722-0876 > Tel: 1-847-491-8163 > Email: iraicu at eecs.northwestern.edu > Web: http://www.eecs.northwestern.edu/~iraicu/ > https://wiki.cucis.eecs.northwestern.edu/ > ================================================================= > ================================================================= > > > ------------------------------------------------------------------------ > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: moz-screenshot-2.jpg Type: image/jpeg Size: 79153 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: moz-screenshot-3.jpg Type: image/jpeg Size: 79671 bytes Desc: not available URL: From foster at anl.gov Tue Oct 27 16:49:15 2009 From: foster at anl.gov (Ian Foster) Date: Tue, 27 Oct 2009 16:49:15 -0500 Subject: [Swift-devel] Swift and BGP plots In-Reply-To: <4AE76018.5090007@cs.uchicago.edu> References: <1256576802.4067.3.camel@localhost> <4AE61640.9080001@cs.uchicago.edu> <1256597086.10196.38.camel@localhost> <4AE674B5.5000406@cs.uchicago.edu> <1256621747.17056.67.camel@localhost> <4AE68820.3050606@cs.uchicago.edu> <1256622669.18682.0.camel@localhost> <1256665502.23397.0.camel@localhost> <4AE7473F.4010404@cs.uchicago.edu> <1256672178.25578.14.camel@localhost> <4AE74E9C.5000604@cs.uchicago.edu> <4AE74F57.5030306@cs.uchicago.edu> <1256673433.26427.0.camel@localhost> <4AE753C4.9070405@cs.uchicago.edu> <1256674591.26911.1.camel@localhost> <4AE75708.8020404@cs.uchicago.edu> <4AE76018.5090007@cs.uchicago.edu> Message-ID: <8243496C-6CA7-46ED-9108-9734296B4D49@anl.gov> great On Oct 27, 2009, at 4:03 PM, Ioan Raicu wrote: > Alright, here it is again: > 4K run: Average Task Execution Time (ms): 61946.37017455522 +/- > 4884.571426229936 > Efficiency: 96.8% > > 6K run: Average Task Execution Time (ms): 64415.056860142206 +/- > 6090.13371098675 > Efficiency: 93.1% > > These both look pretty nice. > > Now, the only thing that still looks odd, is that in the 4K run, I > see 8K active tasks, and in the 6K run I see 12K active tasks. I > also have 128K tasks in total. Are my plots off by 2X somehow? > > Ioan > > > 4K run: > > > 6K run: > > > > Ioan Raicu wrote: >> >> Great! >> >> Mihael Hategan wrote: >>> >>> On Tue, 2009-10-27 at 15:10 -0500, Ioan Raicu wrote: >>> >>>> Which of these should I use in finding the Complete event? Or are >>>> they >>>> equivalent? >>>> >>>> 2009-10-25 21:42:10,717-0500 INFO LateBindingScheduler >>>> Task(type=JOB_SUBMISSION, identity=urn:0-1-12173-1-1-1256524749960) >>>> Completed. Waiting: 3606, Running: 16387. Heap size: 1003M, Heap >>>> free: >>>> 66M, Max heap: 1024M >>>> >>>> 2009-10-25 21:42:10,608-0500 DEBUG TaskImpl Task >>>> (type=JOB_SUBMISSION, >>>> identity=urn:1256524752479-1256524793584-1256524793585) setting >>>> status >>>> to Completed >>>> >>> >>> It's probably best if you used TaskImpl for both Active and >>> Completed. >>> >>> >>> >> >> -- >> ================================================================= >> Ioan Raicu, Ph.D. >> NSF/CRA Computing Innovation Fellow >> ================================================================= >> Center for Ultra-scale Computing and Information Security (CUCIS) >> Department of Electrical Engineering and Computer Science >> Northwestern University >> 2145 Sheridan Rd, Tech M384 >> Evanston, IL 60208-3118 >> ================================================================= >> Cel: 1-847-722-0876 >> Tel: 1-847-491-8163 >> Email: iraicu at eecs.northwestern.edu >> Web: http://www.eecs.northwestern.edu/~iraicu/ >> https://wiki.cucis.eecs.northwestern.edu/ >> ================================================================= >> ================================================================= >> >> >> >> _______________________________________________ >> Swift-devel mailing list >> Swift-devel at ci.uchicago.edu >> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel >> > > -- > ================================================================= > Ioan Raicu, Ph.D. > NSF/CRA Computing Innovation Fellow > ================================================================= > Center for Ultra-scale Computing and Information Security (CUCIS) > Department of Electrical Engineering and Computer Science > Northwestern University > 2145 Sheridan Rd, Tech M384 > Evanston, IL 60208-3118 > ================================================================= > Cel: 1-847-722-0876 > Tel: 1-847-491-8163 > Email: iraicu at eecs.northwestern.edu > Web: http://www.eecs.northwestern.edu/~iraicu/ > https://wiki.cucis.eecs.northwestern.edu/ > ================================================================= > ================================================================= > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From benc at hawaga.org.uk Wed Oct 28 22:49:15 2009 From: benc at hawaga.org.uk (Ben Clifford) Date: Thu, 29 Oct 2009 03:49:15 +0000 (GMT) Subject: [Swift-devel] Swift and BGP plots In-Reply-To: <4AE74E9C.5000604@cs.uchicago.edu> References: <1256576802.4067.3.camel@localhost> <4AE61640.9080001@cs.uchicago.edu> <1256597086.10196.38.camel@localhost> <4AE674B5.5000406@cs.uchicago.edu> <1256621747.17056.67.camel@localhost> <4AE68820.3050606@cs.uchicago.edu> <1256622669.18682.0.camel@localhost> <1256665502.23397.0.camel@localhost> <4AE7473F.4010404@cs.uchicago.edu> <1256672178.25578.14.camel@localhost> <4AE74E9C.5000604@cs.uchicago.edu> Message-ID: you can use swift-plot-log ot do that swift-plot-log my.log karatasks.transitions or something like that. don't reimplement all of that yourself. On Tue, 27 Oct 2009, Ioan Raicu wrote: > OK, so this looks to be a bit more complex then. Can you suggest a way to > extract the Active and Complete timestamps (along with the task ID)? It seems > that if I grep for JOB_SUBMISSION, the status Active is not found on the same > line, so I can't filter according to those two keywords. Also, the Completed > keyword also comes up with many other hits. Any ideas on how to get the info I > need (with cat, grep, etc) without writing some custom script/program that > actually understands in depth the logging of the Swift log? > > Thanks, > Ioan > > Mihael Hategan wrote: > > On Tue, 2009-10-27 at 14:17 -0500, Ioan Raicu wrote: > > > > > OK, I converted the logs, and here is what I got. > > > > > > The thing that bugs me is that things ran faster (per task) at the > > > larger scale (a bit counter intuitive). > > > > > > > Nope. Same workload but more processors. > > > > > Also, the maximum number of concurrent tasks I found was 20K, in both > > > experiments. This doesn't seem right, as the experiments should have > > > 4K or 6K active tasks at any time. > > > > > > > Right. You should see around 4k or 6k max active tasks. > > > > > > > Could I be looking at the wrong entries from the Swift logs? > > > > > > I did this to get the data I needed from the Swift logs: > > > cat dc-4000.log | grep "JOB_START" > dc-4000-start-end.txt > > > cat dc-4000.log | grep "JOB_END" >> dc-4000-start-end.txt > > > > > > Is this correct? Or should I be looking for other job events? > > > > > > > You should probably be looking for "Task(type=JOB_SUBMISSION...) setting > > status to Active" and "Task(type=JOB_SUBMISSION...) setting status to > > Completed". Or something like that. > > > > JOB_START and JOB_END are statuses for the Swift execute2 processes. > > > > > > > > From iraicu at cs.uchicago.edu Wed Oct 28 23:00:55 2009 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Wed, 28 Oct 2009 23:00:55 -0500 Subject: [Swift-devel] Swift and BGP plots In-Reply-To: References: <1256576802.4067.3.camel@localhost> <4AE61640.9080001@cs.uchicago.edu> <1256597086.10196.38.camel@localhost> <4AE674B5.5000406@cs.uchicago.edu> <1256621747.17056.67.camel@localhost> <4AE68820.3050606@cs.uchicago.edu> <1256622669.18682.0.camel@localhost> <1256665502.23397.0.camel@localhost> <4AE7473F.4010404@cs.uchicago.edu> <1256672178.25578.14.camel@localhost> <4AE74E9C.5000604@cs.uchicago.edu> Message-ID: <4AE91377.4090604@cs.uchicago.edu> I managed to do it with a series of grep commands. I did: cat dc-6000.log | grep "JOB_SUBMISSION" | grep "TaskImpl" | grep "Active" > dc-6000-active-completed.txt cat dc-6000.log | grep "JOB_SUBMISSION" | grep "TaskImpl" | grep "Completed" >> dc-6000-active-completed.txt This new parsed log, was then fed through 3 more programs: java ParseSwiftLog2 dc-6000-active-completed.txt > falkon_task_perf.txt java NormalizeTaskPerf falkon_task_perf.txt > falkon_task_perf_normalized.txt java ConvertPerTaskToSummary falkon_task_perf_normalized.txt 1 > falkon_summary.txt and voila, I had the 2 logs Falkon usually generates, the per task log, and the summary log. I could then run: java CompareRuns dc-4000/falkon_task_perf.txt dc-6000/falkon_task_perf.txt 131077 to get basic statistics on the run and comparison between runs. I could also run the standard Falkon plots on these logs as well. Ioan Ben Clifford wrote: > you can use swift-plot-log ot do that > > swift-plot-log my.log karatasks.transitions > > or something like that. > > don't reimplement all of that yourself. > > On Tue, 27 Oct 2009, Ioan Raicu wrote: > > >> OK, so this looks to be a bit more complex then. Can you suggest a way to >> extract the Active and Complete timestamps (along with the task ID)? It seems >> that if I grep for JOB_SUBMISSION, the status Active is not found on the same >> line, so I can't filter according to those two keywords. Also, the Completed >> keyword also comes up with many other hits. Any ideas on how to get the info I >> need (with cat, grep, etc) without writing some custom script/program that >> actually understands in depth the logging of the Swift log? >> >> Thanks, >> Ioan >> >> Mihael Hategan wrote: >> >>> On Tue, 2009-10-27 at 14:17 -0500, Ioan Raicu wrote: >>> >>> >>>> OK, I converted the logs, and here is what I got. >>>> >>>> The thing that bugs me is that things ran faster (per task) at the >>>> larger scale (a bit counter intuitive). >>>> >>>> >>> Nope. Same workload but more processors. >>> >>> >>>> Also, the maximum number of concurrent tasks I found was 20K, in both >>>> experiments. This doesn't seem right, as the experiments should have >>>> 4K or 6K active tasks at any time. >>>> >>>> >>> Right. You should see around 4k or 6k max active tasks. >>> >>> >>> >>>> Could I be looking at the wrong entries from the Swift logs? >>>> >>>> I did this to get the data I needed from the Swift logs: >>>> cat dc-4000.log | grep "JOB_START" > dc-4000-start-end.txt >>>> cat dc-4000.log | grep "JOB_END" >> dc-4000-start-end.txt >>>> >>>> Is this correct? Or should I be looking for other job events? >>>> >>>> >>> You should probably be looking for "Task(type=JOB_SUBMISSION...) setting >>> status to Active" and "Task(type=JOB_SUBMISSION...) setting status to >>> Completed". Or something like that. >>> >>> JOB_START and JOB_END are statuses for the Swift execute2 processes. >>> >>> >>> >>> >> > > -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From iraicu at cs.uchicago.edu Wed Oct 28 23:11:33 2009 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Wed, 28 Oct 2009 23:11:33 -0500 Subject: [Swift-devel] Swift and BGP plots In-Reply-To: <4AE76018.5090007@cs.uchicago.edu> References: <1256576802.4067.3.camel@localhost> <4AE61640.9080001@cs.uchicago.edu> <1256597086.10196.38.camel@localhost> <4AE674B5.5000406@cs.uchicago.edu> <1256621747.17056.67.camel@localhost> <4AE68820.3050606@cs.uchicago.edu> <1256622669.18682.0.camel@localhost> <1256665502.23397.0.camel@localhost> <4AE7473F.4010404@cs.uchicago.edu> <1256672178.25578.14.camel@localhost> <4AE74E9C.5000604@cs.uchicago.edu> <4AE74F57.5030306@cs.uchicago.edu> <1256673433.26427.0.camel@localhost> <4AE753C4.9070405@cs.uchicago.edu> <1256674591.26911.1.camel@localhost> <4AE75708.8020404@cs.uchicago.edu> <4AE76018.5090007@cs.uchicago.edu> Message-ID: <4AE915F5.8070208@cs.uchicago.edu> Mihael, Did you figure out why I am seeing 8K and 12K active tasks, when we only had 4K and 6K CPU cores? Were there really 128K tasks in the workflow? Just want to make sure the log conversion worked as expected. Also, assuming there were really 128K tasks of 60 sec each, and 8K CPUs, the ideal time to complete the run 4K would be 960 sec. Run4K ran in 1183 sec, giving us an end-to-end efficiency of 81%. For the run6K, the ideal time was 640 sec, so with an actual time of 884, we got an end-to-end efficiency of 72%. Not quite the 90%+ efficiencies when looking at a per task level, but still quite good! Ioan Ioan Raicu wrote: > Alright, here it is again: > 4K run: Average Task Execution Time (ms): 61946.37017455522 +/- > 4884.571426229936 > Efficiency: 96.8% > > 6K run: Average Task Execution Time (ms): 64415.056860142206 +/- > 6090.13371098675 > Efficiency: 93.1% > > These both look pretty nice. > > Now, the only thing that still looks odd, is that in the 4K run, I see > 8K active tasks, and in the 6K run I see 12K active tasks. I also have > 128K tasks in total. Are my plots off by 2X somehow? > > Ioan > > > 4K run: > > > 6K run: > > > > Ioan Raicu wrote: >> Great! >> >> Mihael Hategan wrote: >>> On Tue, 2009-10-27 at 15:10 -0500, Ioan Raicu wrote: >>> >>>> Which of these should I use in finding the Complete event? Or are they >>>> equivalent? >>>> >>>> 2009-10-25 21:42:10,717-0500 INFO LateBindingScheduler >>>> Task(type=JOB_SUBMISSION, identity=urn:0-1-12173-1-1-1256524749960) >>>> Completed. Waiting: 3606, Running: 16387. Heap size: 1003M, Heap free: >>>> 66M, Max heap: 1024M >>>> >>>> 2009-10-25 21:42:10,608-0500 DEBUG TaskImpl Task(type=JOB_SUBMISSION, >>>> identity=urn:1256524752479-1256524793584-1256524793585) setting status >>>> to Completed >>>> >>> >>> It's probably best if you used TaskImpl for both Active and Completed. >>> >>> >>> >> >> -- >> ================================================================= >> Ioan Raicu, Ph.D. >> NSF/CRA Computing Innovation Fellow >> ================================================================= >> Center for Ultra-scale Computing and Information Security (CUCIS) >> Department of Electrical Engineering and Computer Science >> Northwestern University >> 2145 Sheridan Rd, Tech M384 >> Evanston, IL 60208-3118 >> ================================================================= >> Cel: 1-847-722-0876 >> Tel: 1-847-491-8163 >> Email: iraicu at eecs.northwestern.edu >> Web: http://www.eecs.northwestern.edu/~iraicu/ >> https://wiki.cucis.eecs.northwestern.edu/ >> ================================================================= >> ================================================================= >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> Swift-devel mailing list >> Swift-devel at ci.uchicago.edu >> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel >> > > -- > ================================================================= > Ioan Raicu, Ph.D. > NSF/CRA Computing Innovation Fellow > ================================================================= > Center for Ultra-scale Computing and Information Security (CUCIS) > Department of Electrical Engineering and Computer Science > Northwestern University > 2145 Sheridan Rd, Tech M384 > Evanston, IL 60208-3118 > ================================================================= > Cel: 1-847-722-0876 > Tel: 1-847-491-8163 > Email: iraicu at eecs.northwestern.edu > Web: http://www.eecs.northwestern.edu/~iraicu/ > https://wiki.cucis.eecs.northwestern.edu/ > ================================================================= > ================================================================= > > -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 79153 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 79671 bytes Desc: not available URL: From hategan at mcs.anl.gov Thu Oct 29 00:44:37 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 29 Oct 2009 00:44:37 -0500 Subject: [Swift-devel] Swift and BGP plots In-Reply-To: <4AE915F5.8070208@cs.uchicago.edu> References: <1256576802.4067.3.camel@localhost> <4AE61640.9080001@cs.uchicago.edu> <1256597086.10196.38.camel@localhost> <4AE674B5.5000406@cs.uchicago.edu> <1256621747.17056.67.camel@localhost> <4AE68820.3050606@cs.uchicago.edu> <1256622669.18682.0.camel@localhost> <1256665502.23397.0.camel@localhost> <4AE7473F.4010404@cs.uchicago.edu> <1256672178.25578.14.camel@localhost> <4AE74E9C.5000604@cs.uchicago.edu> <4AE74F57.5030306@cs.uchicago.edu> <1256673433.26427.0.camel@localhost> <4AE753C4.9070405@cs.uchicago.edu> <1256674591.26911.1.camel@localhost> <4AE75708.8020404@cs.uchicago.edu> <4AE76018.5090007@cs.uchicago.edu> <4AE915F5.8070208@cs.uchicago.edu> Message-ID: <1256795077.20508.16.camel@localhost> On Wed, 2009-10-28 at 23:11 -0500, Ioan Raicu wrote: > Mihael, > Did you figure out why I am seeing 8K and 12K active tasks, when we > only had 4K and 6K CPU cores? Haven't tried. > Were there really 128K tasks in the workflow? Nope. 64K. > Just want to make sure the log conversion worked as expected. > > Also, assuming there were really 128K tasks of 60 sec each, and 8K > CPUs, the ideal time to complete the run 4K would be 960 sec. That's one calculation that won't be bothered by doubling everything. But no, there were 64k tasks. > Run4K ran in 1183 sec, giving us an end-to-end efficiency of 81%. > > For the run6K, the ideal time was 640 sec, so with an actual time of > 884, we got an end-to-end efficiency of 72%. It depends whether you count from the time the partition boots or from the time swift starts. We could count the queue/partition boot time, but that doesn't tell us much about swift. On the other hand, if we don't there's still some submission happening during that time, so that counts. The numbers for Falkon, were the workers started already? > > Not quite the 90%+ efficiencies when looking at a per task level, but > still quite good! I'm not quite sure what's happening. Maybe I wasn't clear. Though I was. Is there some misunderstanding here about the different things being measured and how? From iraicu at cs.uchicago.edu Thu Oct 29 09:30:00 2009 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Thu, 29 Oct 2009 09:30:00 -0500 Subject: [Swift-devel] Swift and BGP plots In-Reply-To: <1256795077.20508.16.camel@localhost> References: <1256576802.4067.3.camel@localhost> <4AE61640.9080001@cs.uchicago.edu> <1256597086.10196.38.camel@localhost> <4AE674B5.5000406@cs.uchicago.edu> <1256621747.17056.67.camel@localhost> <4AE68820.3050606@cs.uchicago.edu> <1256622669.18682.0.camel@localhost> <1256665502.23397.0.camel@localhost> <4AE7473F.4010404@cs.uchicago.edu> <1256672178.25578.14.camel@localhost> <4AE74E9C.5000604@cs.uchicago.edu> <4AE74F57.5030306@cs.uchicago.edu> <1256673433.26427.0.camel@localhost> <4AE753C4.9070405@cs.uchicago.edu> <1256674591.26911.1.camel@localhost> <4AE75708.8020404@cs.uchicago.edu> <4AE76018.5090007@cs.uchicago.edu> <4AE915F5.8070208@cs.uchicago.edu> <1256795077.20508.16.camel@localhost> Message-ID: <4AE9A6E8.60609@cs.uchicago.edu> Mihael Hategan wrote: > On Wed, 2009-10-28 at 23:11 -0500, Ioan Raicu wrote: > >> Mihael, >> Did you figure out why I am seeing 8K and 12K active tasks, when we >> only had 4K and 6K CPU cores? >> > > Haven't tried. > > >> Were there really 128K tasks in the workflow? >> > > Nope. 64K. > OK, it would be good to look at why we have double the # of tasks. It must be my filtering of the Swift log. Here was my filtered log: http://www.ece.northwestern.edu/~iraicu/scratch/logs/dc-4000-active-completed.txt This filtered log was generated by: cat dc-4000.log | grep "JOB_SUBMISSION" | grep "TaskImpl" | grep "Active" > dc-4000-active-completed.txt cat dc-4000.log | grep "JOB_SUBMISSION" | grep "TaskImpl" | grep "Completed" >> dc-4000-active-completed.txt > >> Just want to make sure the log conversion worked as expected. >> >> Also, assuming there were really 128K tasks of 60 sec each, and 8K >> CPUs, the ideal time to complete the run 4K would be 960 sec. >> > > That's one calculation that won't be bothered by doubling everything. > But no, there were 64k tasks. > > If there were 64K tasks and 4K CPUs, then the ideal time will be the same, 960 sec. >> Run4K ran in 1183 sec, giving us an end-to-end efficiency of 81%. >> >> For the run6K, the ideal time was 640 sec, so with an actual time of >> 884, we got an end-to-end efficiency of 72%. >> > > It depends whether you count from the time the partition boots or from > the time swift starts. We could count the queue/partition boot time, but > that doesn't tell us much about swift. On the other hand, if we don't > there's still some submission happening during that time, so that > counts. > I count from where the log starts. There is about 20 seconds of inactivity at the beginning of the log, but at around 20 sec in one log, and 24 sec in the other log, 1 job is submitted and running. At about 120 second into the run, the floodgate is opened and many jobs are submitted and start running. So, should we count from time 0, 20, or 120? I guess its all about what you are trying to measure and show. In all cases, I think the workers were provisioned, its just a matter of how much of the Swift overhead you want to take into account I think. > The numbers for Falkon, were the workers started already? > > Yes, the workers were already provisioned in that case. >> Not quite the 90%+ efficiencies when looking at a per task level, but >> still quite good! >> > > I'm not quite sure what's happening. Maybe I wasn't clear. Though I was. > Is there some misunderstanding here about the different things being > measured and how? > No. The real way to compute efficiency is to use the end-to-end time of the real run compared to the ideal run. The other efficiency I sometimes throw out is the per task efficiency, where you take the average real run time of all tasks, and compare it to the ideal time of a task. This second measure of efficiency is usually optimistic, but it allows us to measure efficiency between various different runs that might be too difficult to compare using the traditional efficiency metric. Ioan > > -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Thu Oct 29 10:17:24 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 29 Oct 2009 10:17:24 -0500 Subject: [Swift-devel] Swift and BGP plots In-Reply-To: <4AE9A6E8.60609@cs.uchicago.edu> References: <1256576802.4067.3.camel@localhost> <4AE61640.9080001@cs.uchicago.edu> <1256597086.10196.38.camel@localhost> <4AE674B5.5000406@cs.uchicago.edu> <1256621747.17056.67.camel@localhost> <4AE68820.3050606@cs.uchicago.edu> <1256622669.18682.0.camel@localhost> <1256665502.23397.0.camel@localhost> <4AE7473F.4010404@cs.uchicago.edu> <1256672178.25578.14.camel@localhost> <4AE74E9C.5000604@cs.uchicago.edu> <4AE74F57.5030306@cs.uchicago.edu> <1256673433.26427.0.camel@localhost> <4AE753C4.9070405@cs.uchicago.edu> <1256674591.26911.1.camel@localhost> <4AE75708.8020404@cs.uchicago.edu> <4AE76018.5090007@cs.uchicago.edu> <4AE915F5.8070208@cs.uchicago.edu> <1256795077.20508.16.camel@localhost> <4AE9A6E8.60609@cs.uchicago.edu> Message-ID: <1256829444.21206.9.camel@localhost> On Thu, 2009-10-29 at 09:30 -0500, Ioan Raicu wrote: > > Nope. 64K. > > > OK, it would be good to look at why we have double the # of tasks. It > must be my filtering of the Swift log. Here was my filtered log: > http://www.ece.northwestern.edu/~iraicu/scratch/logs/dc-4000-active-completed.txt Both the coaster service log and swift log go to the same place in that case. You'll see a difference in the way the task IDs look. That's something you can use. This is a task on the swift side: identity=urn:0-1-11475-1-1-1256524749943 This is on the coaster side: identity=urn:1256524750479-1256524791529-1256524791530 [...] > > > > It depends whether you count from the time the partition boots or from > > the time swift starts. We could count the queue/partition boot time, but > > that doesn't tell us much about swift. On the other hand, if we don't > > there's still some submission happening during that time, so that > > counts. > > > I count from where the log starts. There is about 20 seconds of > inactivity at the beginning of the log, but at around 20 sec in one > log, and 24 sec in the other log, 1 job is submitted and running. That's the worker block. It's "running" in that cobalt says so, but it's booting. > At about 120 second into the run, the floodgate is opened and many > jobs are submitted and start running. So, should we count from time 0, > 20, or 120? 100. The 20 seconds of activity in the beginning are real swift/submission overhead. The waiting until the partition boots isn't. > I guess its all about what you are trying to measure and show. In all > cases, I think the workers were provisioned, its just a matter of how > much of the Swift overhead you want to take into account I think. > > The numbers for Falkon, were the workers started already? > > > > > Yes, the workers were already provisioned in that case. > > > Not quite the 90%+ efficiencies when looking at a per task level, but > > > still quite good! > > > > > > > I'm not quite sure what's happening. Maybe I wasn't clear. Though I was. > > Is there some misunderstanding here about the different things being > > measured and how? > > > No. The real way to compute efficiency is to use the end-to-end time > of the real run compared to the ideal run. The other efficiency I > sometimes throw out is the per task efficiency, where you take the > average real run time of all tasks, and compare it to the ideal time > of a task. This second measure of efficiency is usually optimistic, > but it allows us to measure efficiency between various different runs > that might be too difficult to compare using the traditional > efficiency metric. Again, I believe the latter to be arbitrary. That's because according to it you can have very low efficiencies yet linear speedups. In addition, I see no literature to use it. If you want to use such an arbitrary measure, fine. Please don't use it on this. From iraicu at cs.uchicago.edu Thu Oct 29 12:04:00 2009 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Thu, 29 Oct 2009 12:04:00 -0500 Subject: [Swift-devel] Swift and BGP plots In-Reply-To: <1256829444.21206.9.camel@localhost> References: <1256576802.4067.3.camel@localhost> <4AE61640.9080001@cs.uchicago.edu> <1256597086.10196.38.camel@localhost> <4AE674B5.5000406@cs.uchicago.edu> <1256621747.17056.67.camel@localhost> <4AE68820.3050606@cs.uchicago.edu> <1256622669.18682.0.camel@localhost> <1256665502.23397.0.camel@localhost> <4AE7473F.4010404@cs.uchicago.edu> <1256672178.25578.14.camel@localhost> <4AE74E9C.5000604@cs.uchicago.edu> <4AE74F57.5030306@cs.uchicago.edu> <1256673433.26427.0.camel@localhost> <4AE753C4.9070405@cs.uchicago.edu> <1256674591.26911.1.camel@localhost> <4AE75708.8020404@cs.uchicago.edu> <4AE76018.5090007@cs.uchicago.edu> <4AE915F5.8070208@cs.uchicago.edu> <1256795077.20508.16.camel@localhost> <4AE9A6E8.60609@cs.uchicago.edu> <1256829444.21206.9.camel@localhost> Message-ID: <4AE9CB00.9020808@cs.uchicago.edu> hmmm, any recommendation on how to parse them out from each other? My simple cat and grep probably won't work. Is there a patterns at least, on the number of dashes "-"? Ioan Mihael Hategan wrote: > On Thu, 2009-10-29 at 09:30 -0500, Ioan Raicu wrote: > > >>> Nope. 64K. >>> >>> >> OK, it would be good to look at why we have double the # of tasks. It >> must be my filtering of the Swift log. Here was my filtered log: >> http://www.ece.northwestern.edu/~iraicu/scratch/logs/dc-4000-active-completed.txt >> > > Both the coaster service log and swift log go to the same place in that > case. You'll see a difference in the way the task IDs look. That's > something you can use. > This is a task on the swift side: > identity=urn:0-1-11475-1-1-1256524749943 > This is on the coaster side: > identity=urn:1256524750479-1256524791529-1256524791530 > [...] > >>> It depends whether you count from the time the partition boots or from >>> the time swift starts. We could count the queue/partition boot time, but >>> that doesn't tell us much about swift. On the other hand, if we don't >>> there's still some submission happening during that time, so that >>> counts. >>> >>> >> I count from where the log starts. There is about 20 seconds of >> inactivity at the beginning of the log, but at around 20 sec in one >> log, and 24 sec in the other log, 1 job is submitted and running. >> > > That's the worker block. It's "running" in that cobalt says so, but it's > booting. > > >> At about 120 second into the run, the floodgate is opened and many >> jobs are submitted and start running. So, should we count from time 0, >> 20, or 120? >> > > 100. The 20 seconds of activity in the beginning are real > swift/submission overhead. The waiting until the partition boots isn't. > > >> I guess its all about what you are trying to measure and show. In all >> cases, I think the workers were provisioned, its just a matter of how >> much of the Swift overhead you want to take into account I think. >> >>> The numbers for Falkon, were the workers started already? >>> >>> >>> >> Yes, the workers were already provisioned in that case. >> >>>> Not quite the 90%+ efficiencies when looking at a per task level, but >>>> still quite good! >>>> >>>> >>> I'm not quite sure what's happening. Maybe I wasn't clear. Though I was. >>> Is there some misunderstanding here about the different things being >>> measured and how? >>> >>> >> No. The real way to compute efficiency is to use the end-to-end time >> of the real run compared to the ideal run. The other efficiency I >> sometimes throw out is the per task efficiency, where you take the >> average real run time of all tasks, and compare it to the ideal time >> of a task. This second measure of efficiency is usually optimistic, >> but it allows us to measure efficiency between various different runs >> that might be too difficult to compare using the traditional >> efficiency metric. >> > > Again, I believe the latter to be arbitrary. That's because according to > it you can have very low efficiencies yet linear speedups. In addition, > I see no literature to use it. > > If you want to use such an arbitrary measure, fine. Please don't use it > on this. > > > -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Thu Oct 29 13:50:47 2009 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 29 Oct 2009 13:50:47 -0500 Subject: [Swift-devel] Swift and BGP plots In-Reply-To: <4AE9CB00.9020808@cs.uchicago.edu> References: <1256576802.4067.3.camel@localhost> <4AE61640.9080001@cs.uchicago.edu> <1256597086.10196.38.camel@localhost> <4AE674B5.5000406@cs.uchicago.edu> <1256621747.17056.67.camel@localhost> <4AE68820.3050606@cs.uchicago.edu> <1256622669.18682.0.camel@localhost> <1256665502.23397.0.camel@localhost> <4AE7473F.4010404@cs.uchicago.edu> <1256672178.25578.14.camel@localhost> <4AE74E9C.5000604@cs.uchicago.edu> <4AE74F57.5030306@cs.uchicago.edu> <1256673433.26427.0.camel@localhost> <4AE753C4.9070405@cs.uchicago.edu> <1256674591.26911.1.camel@localhost> <4AE75708.8020404@cs.uchicago.edu> <4AE76018.5090007@cs.uchicago.edu> <4AE915F5.8070208@cs.uchicago.edu> <1256795077.20508.16.camel@localhost> <4AE9A6E8.60609@cs.uchicago.edu> <1256829444.21206.9.camel@localhost> <4AE9CB00.9020808@cs.uchicago.edu> Message-ID: <1256842247.23667.0.camel@localhost> On Thu, 2009-10-29 at 12:04 -0500, Ioan Raicu wrote: > hmmm, any recommendation on how to parse them out from each other? My > simple cat and grep probably won't work. Is there a patterns at least, > on the number of dashes "-"? ".*:0-.*" will give you the swift tasks. > > This is a task on the swift side: > > identity=urn:0-1-11475-1-1-1256524749943 > > This is on the coaster side: > > identity=urn:1256524750479-1256524791529-1256524791530 From benc at hawaga.org.uk Sat Oct 10 18:59:30 2009 From: benc at hawaga.org.uk (Ben Clifford) Date: Sat, 10 Oct 2009 23:59:30 -0000 Subject: [Swift-devel] [provenance-challenge] CFP: FGCS Special Issue on Using the Open Provenance Model to Address Interoperability Challenges (fwd) Message-ID: ---------- Forwarded message ---------- Date: Fri, 9 Oct 2009 15:13:42 +0000 From: Yogesh Simmhan Reply-To: provenance-challenge at ipaw.info To: "provenance-challenge at ipaw.info" Subject: [provenance-challenge] CFP: FGCS Special Issue on Using the Open Provenance Model to Address Interoperability Challenges This is the special issue on OPM that was mentioned at the Provenance Challenge workshop. We encourage you to submit an article to this issue since it will try and capture all the work going on around the OPM effort. If you plan on submitting an article, please drop an email to one of the editors letting us know of your intent. This will help us plan accordingly. Regards, --Yogesh, Paul & Luc Future Generation Computer Systems The International Journal of Grid Computing and eScience Special Issue on Using the Open Provenance Model to Address Interoperability Challenges Guest Editors: Yogesh Simmhan, Paul Groth, Luc Moreau June 29, 2009 The Journal Future Generation Computer Systems invites authors to submit papers for the Special Issue on Using the Open Provenance Model to Address Interoperability Challenges. This special issue follows the Provenance Challenge 3 workshop on the same topic, but is open also to contributions by teams that were not represented at the workshop. (1) Background Data products are increasingly being produced and "mashed-up" by the composition of services and data supplied by multiple parties using a variety of databases, data analysis, management, and collection technologies. This approach is particular evident in e-Science where scientists combine sensor data and shared Web-accessible databases using a variety of local and remote data analysis routines to produce experimental results, which they published and get reused by other scientists. In such environments, provenance (also referred to as audit trail, lineage, and pedigree) plays a critical role as it enables users to understand, verify, reproduce, and ascertain the quality of data products. An important challenge in the context of these compositional applications is how to integrate the provenance data produced by different systems to be able to construct the full provenance of complex data products across the different systems involved in their derivation. To that end, a common data model for provenance, the Open Provenance Model (OPM), was proposed to help ease the integration of provenance data across the heterogeneous environments used for running such applications. To evaluate the suitability of OPM and gain practical experience with interoperability issues related to provenance across heterogeneous systems, 14 teams from across the world have participated in the Third Provenance Challenge since March 2nd. During this challenge, teams have exchanged provenance data between their provenance systems using OPM. They have developed OPM serializations in both RDF and XML, ran provenance queries over the exchanged data, and began to create common tools for use with OPM. (2) Topics The aim of the special issue is to provide an archival view of the state-of-the-art of both practical and theoretical issues related to provenance interoperability across systems using the Open Provenance Model. We therefore encourage submissions in the following areas: * Theoretical considerations pertaining to the Open Provenance Model * Semantic inter-operability with the Open Provenance Model * System issues and OPM (including integration, performance, storage) * Provenance models and comparisons with OPM * Real applications making use of OPM; usability of provenance * Open Provenance Model and databases * Domain specific specialisations of the Open Provenance Model * Query languages for OPM * Interoperable queries over the Open Provenance Model * Provenance tools that use OPM (3) Instructions to the authors Two types of submissions are permitted: * short papers should have a maximum length of 6 pages, whereas * long papers should be limited to 12 pages. Articles should be submitted electronically via the journal's online submission and peer-review systems at http://ees.elsevier.com/fgcs/. LATEX and Word format are acceptable. Formatting instructions are available from the submission page. (4) Tentative Schedule 1. Submission deadline: December 15 2009 2. Notification of acceptance: March 30 2010 3. Camera ready version: May 15 2010 The schedule may be subject to revisions. Prospective authors are invited to make themselves known to the editors ahead of time to facilitate the harmonization of the issue and ensure that the authors will be informed of any change. From benc at hawaga.org.uk Sun Oct 18 21:00:03 2009 From: benc at hawaga.org.uk (Ben Clifford) Date: Mon, 19 Oct 2009 02:00:03 -0000 Subject: [Swift-devel] re: IO overheads of swift wrapper scripts on BlueGene/P Message-ID: swift r2970 changed copying outputs to a hard-link of outputs to improve performance at the output stage. you might be interested to see how that works. (also, don't use the same runid for different runs. you'll get into trouble somehow sooner or later. ) r2970 | benc | 2009-06-22 23:40:53 +1000 (Mon, 22 Jun 2009) | 6 lines Make wrapper use mv to stage out files to shared directory rather than cp. This should be faster when the job directories are on the same filesystem as the site shared cache. (suggested by tuecke) -- From benc at hawaga.org.uk Mon Oct 19 00:49:31 2009 From: benc at hawaga.org.uk (Ben Clifford) Date: Mon, 19 Oct 2009 05:49:31 -0000 Subject: [Swift-devel] [Fwd: re: IO overheads of swift wrapper scripts on BlueGene/P (fwd)] In-Reply-To: <1255930756.7690.7.camel@localhost> References: <1255930330.7690.0.camel@localhost> <1255930756.7690.7.camel@localhost> Message-ID: On Mon, 19 Oct 2009, Mihael Hategan wrote: > Right. We're already using that. ok - I was comparing version numbers in Allan's initial mail which suggested not. > mv, on the same fs, would just make an > entry in the new directory pointing to the same data instead of > duplicating the data. Though I suppose this depends on the FS. I guess any posixy file system would work ok. Makes me wonder how ln (and hard-links in general) behave(s) on an MS-DOS filesystem mounted under linux, thoug. (not that I'm suggesting that's a sensible place to run stuff) -- From benc at hawaga.org.uk Mon Oct 19 01:45:42 2009 From: benc at hawaga.org.uk (Ben Clifford) Date: Mon, 19 Oct 2009 06:45:42 -0000 Subject: [Swift-devel] [Fwd: re: IO overheads of swift wrapper scripts on BlueGene/P (fwd)] In-Reply-To: <1255930756.7690.7.camel@localhost> References: <1255930330.7690.0.camel@localhost> <1255930756.7690.7.camel@localhost> Message-ID: On Mon, 19 Oct 2009, Mihael Hategan wrote: > On this topic, why did we unsubscribe one of swift's committers from the > swift mailing list(s)? I think that's part of the general shutdown of my CI account (the same process that took me off the secret internal list). Easy enough for me to resubscribe, I guess. -- From iraicu at eecs.northwestern.edu Mon Oct 26 16:57:46 2009 From: iraicu at eecs.northwestern.edu (Ioan Raicu) Date: Mon, 26 Oct 2009 21:57:46 -0000 Subject: [Swift-devel] Swift and BGP In-Reply-To: <1256589386.7792.23.camel@localhost> References: <1256576209.1135.46.camel@localhost> <924B1323-386B-4E2C-8B97-B0C54E6FC1FB@anl.gov> <1256589386.7792.23.camel@localhost> Message-ID: <4AE619FC.4090402@eecs.northwestern.edu> Although, I don't know if we understand the Swift limits all that well. We never managed to do a systematic performance evaluation of Swift (with the exception of the initial 2007 results from the SWF07 paper), or at least I am not aware of it. Perhaps this is a good excuse to do such a performance evaluation. Ioan -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= Mihael Hategan wrote: > On Mon, 2009-10-26 at 15:07 -0500, Ian Foster wrote: > >> Hi Mihael: >> >> This is very encouraging. >> >> It will be helpful to understand how these numbers compare to Falkon, >> as Falkon is the one other data point we have on what can be achieved >> on BGP. >> > > Sure. Though I suspect swift limits are reached before falkon or coaster > limits are reached. So unless we compare raw falkon and raw coasters, we > probably won't see much of a difference. Which would then be pretty > useless for Swift. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From npatel94 at uncc.edu Thu Oct 8 12:40:57 2009 From: npatel94 at uncc.edu (nirali patel) Date: Thu, 08 Oct 2009 17:40:57 -0000 Subject: [Swift-devel] Unable to destroy remote service for task Message-ID: <3f8c8bba0910081033w46d39aa9md86a52573873e8c6@mail.gmail.com> Hello I am trying to test Proxy delegation in Grid using Java Cog. Let me explain in short how it works.Main program is running on server grid02. which submits job to server grid03. In turn server grid03 will submit a job to grid04. I have created proxy only on server grid02. So, server 02 delegates proxy to server grid03 ans server grid03 in turn delegates proxy to grid04 . But when I run this Main program, job is submitted properly. But next server grid03 fails to submit a job on server grid04. It gives me following exception. - 1255023074894 Status: Failed - Unable to destroy remote service for task urn:cog-1255023063944 java.lang.NullPointerException at org.globus.exec.generated.service.ManagedJobServiceAddressingLocator.getManagedJobPortTypePort(ManagedJobServiceAddressingLocator.java:12) at org.globus.exec.utils.client.ManagedJobClientHelper.getPort(ManagedJobClientHelper.java:32) at org.globus.exec.client.GramJob.release(GramJob.java:1428) at org.globus.cog.abstraction.impl.execution.gt4_0_0.JobSubmissionTaskHandler.cleanup(JobSubmissionTaskHandler.java:324) at org.globus.cog.abstraction.impl.execution.gt4_0_0.JobSubmissionTaskHandler.submit(JobSubmissionTaskHandler.java:179) at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.submit(AbstractTaskHandler.java:69) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.globus.cog.abstraction.impl.common.sandbox.Sandbox$2.run(Sandbox.java:65) at org.globus.cog.abstraction.impl.common.sandbox.Sandbox.wrap(Sandbox.java:88) at org.globus.cog.abstraction.impl.common.sandbox.Sandbox.invoke(Sandbox.java:61) at org.globus.cog.abstraction.impl.common.sandbox.SandboxingTaskHandler.submit(SandboxingTaskHandler.java:44) at Secondjob.submitJob(Secondjob.java:81) at Secondjob.main(Secondjob.java:29) -- Thank you for your time and concern Nirali -------------- next part -------------- An HTML attachment was scrubbed... URL: