From skrieder at iit.edu Fri May 2 14:50:32 2014 From: skrieder at iit.edu (Scott Krieder) Date: Fri, 2 May 2014 14:50:32 -0500 Subject: [Swift-devel] Google Compute Engine Message-ID: https://github.com/yadudoc/swift-on-cloud If you clone that repo and move into compute-engine there is a readme for Yadu's interface to gce. -Scott -- Scott J. Krieder C: 419-685-0410 E: skrieder at iit.edu http://datasys.cs.iit.edu/~skrieder/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From yadudoc1729 at gmail.com Fri May 2 15:34:43 2014 From: yadudoc1729 at gmail.com (Yadu Nand) Date: Fri, 2 May 2014 15:34:43 -0500 Subject: [Swift-devel] Google Compute Engine In-Reply-To: References: Message-ID: Hi Everyone, Currently I haven't found the proper way to share images publicly and so I have sent out invitations to Dan, Mihael, Justin, Tim, Ketan, Scott and have made Mike and Ketan owners who can manage new members of the project which is hosting the images. Thanks, Yadu On Fri, May 2, 2014 at 2:50 PM, Scott Krieder wrote: > https://github.com/yadudoc/swift-on-cloud > > If you clone that repo and move into compute-engine there is a readme for > Yadu's interface to gce. > > -Scott > > -- > Scott J. Krieder > C: 419-685-0410 > E: skrieder at iit.edu > http://datasys.cs.iit.edu/~skrieder/ > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > -- Yadu Nand B -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Sun May 4 14:29:00 2014 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sun, 4 May 2014 12:29:00 -0700 Subject: [Swift-devel] Ben's view Message-ID: <1399231740.20037.1.camel@echo> A somewhat familiar debugging strategy for javascript: http://alltom.com/pages/theseus/ http://alltom.com/pages/theseus/chi-2014/addressing-misconceptions-theseus.pdf Mihael From iraicu at cs.iit.edu Thu May 8 21:28:13 2014 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Thu, 08 May 2014 21:28:13 -0500 Subject: [Swift-devel] ACM HPDC 2014 -- Early Bird Registration is due May 15th Message-ID: <536C3D3D.2040109@cs.iit.edu> **** CALL FOR PAPERS **** The 23nd International ACM Symposium on High-Performance Parallel and Distributed Computing (HPDC-2014) Vancouver, Canada - June 23-27, 2014 http://www.hpdc.org/2014 The ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) is the premier annual conference for presenting the latest research on the design, implementation, evaluation, and application of parallel and distributed systems for high-end computing. In 2014, the 23nd HPDC and affiliated workshops will take place in the beautiful city of Vancouver, Canada during June 23-27, 2014. **** IMPORTANT DATES **** Early Bird Registration: May 15th, 2014 Workshops: June 23-24, 2014 Main Conference: June 25-27, 2014 Registration Website HPDC'14 registration is now open: https://www.regonline.com/hpdc14 What's Included For all registrant types, registration for the HPDC'14 conference includes the reception on Wednesday and the social event and banquet on Thursday. HPDC registrants can attend the main conference and workshops. There is a separate workshops only registration. Early Registration Deadline The early registration deadline is May 15, 2014 (CET). Early Registration Fees Conference + Workshops (ACM members): $700 Conference + Workshops (non-ACM members): $830 Conference + Workshops (students): $400 Workshops Only (all registrant types): $300 Late Registration Fees For late registration, add $150 to each of the early registration fees (except student-only registration). Visa Support Letters Attendees needing visa letters may request a letter via the ACM. -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Editor: IEEE TCC, Springer Cluster, Springer JoCCASA Chair: IEEE/ACM MTAGS, ACM ScienceCloud ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ LinkedIn: http://www.linkedin.com/in/ioanraicu Google: http://scholar.google.com/citations?user=jE73HYAAAAAJ ================================================================= ================================================================= From iraicu at cs.iit.edu Thu May 8 21:39:20 2014 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Thu, 08 May 2014 21:39:20 -0500 Subject: [Swift-devel] Call for Posters: ACM HPDC 2014 -- due May 16th Message-ID: <536C3FD8.7090205@cs.iit.edu> Call for Posters HPDC'14 will feature a poster session that will provide the right environment for lively and informal discussions on various high performance parallel and distributed computing topics. The poster session will be held on Wednesday, June 25, in the late afternoon. Participating posters will be selected based on the following criteria: - Submissions must describe new, interesting ideas on any HPDC topics of interest - Submissions can present work in progress, and we strongly encourage the authors to include preliminary experimental results, if available - Student submissions meeting the above criteria will be given preference We invite all potential authors to submit their contribution to this poster session in the form of a two-page PDF extended abstract (we recommend using the ACM Proceedings style, and fonts not smaller than 10 point). Please provide the following information in your PDF file: - Poster title - Author names, affiliations, and email addresses - Note which authors, if any, are students Abstracts must be submitted through email with the Subject "HPDC 2014 Poster Submission" to chandra AT cs DOT umn DOT edu before May 16 2014, 5:00pm EDT. Authors will be notified of acceptance or rejection via e-mail by May 23, 2014. No reviews will be provided. Accepted posters will be published online on the conference website. Details about the poster presentation (e.g., poster size) will be available closer to the conference. For any questions about the submission, selection, and presentation of the accepted posters, please contact the Posters Chair, Abhishek Chandra, University of Minnesota (email: chandra AT cs DOT umn DOT edu), or see http://www.hpdc.org/2014/posters/call-for-posters/ for more information. -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Editor: IEEE TCC, Springer Cluster, Springer JoCCASA Chair: IEEE/ACM MTAGS, ACM ScienceCloud ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ LinkedIn: http://www.linkedin.com/in/ioanraicu Google: http://scholar.google.com/citations?user=jE73HYAAAAAJ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From wilde at anl.gov Wed May 21 11:47:47 2014 From: wilde at anl.gov (Michael Wilde) Date: Wed, 21 May 2014 11:47:47 -0500 Subject: [Swift-devel] failing test case: Re: Multiple output files In-Reply-To: <1395972673.28720.13.camel@echo> References: <532A305B.4000600@anl.gov> <532B9B8C.7000007@anl.gov> <3FD5A438-74FE-418C-822D-229125EA6CDD@uchicago.edu> <532C3B7C.30609@anl.gov> <532C410C.5080309@anl.gov> <532DBAF8.9030300@anl.gov> <1395513490.1382.10.camel@echo> <532DDFA8.3040909@anl.gov> <1395676018.18549.13.camel@echo> <1395858271.26097.5.camel@echo> <53331F9F.9040404@anl.gov> <1395860669.26428.2.camel@echo> <1395972673.28720.13.camel@echo> Message-ID: <537CD8B3.10306@anl.gov> Mihael, in trunk, for your example below: if one does not put the output files in a subdir, and omits ther location arge form the simple_mapper (or specifies location=".") then Swift fails with the following error: swift$ more mc5.swift cs.sh :::::::::::::: mc5.swift :::::::::::::: type file; app (file pSet[] ) genParamSet (file script) { sh @script; } file cs <"cs.sh">; file pSet[] ; pSet = genParamSet(cs); foreach f, i in pSet { string p = readData(f); tracef("%i %s %s\n", i, filename(f), p); } :::::::::::::: cs.sh :::::::::::::: #!/bin/bash echo "p1=01; p2=02" > pf_0000.txt echo "p1=21; p2=22" > pf_0001.txt echo "p1=31; p2=32" > pf_0002.txt swift$ swift mc5.swift Swift trunk swift-r7879 cog-r3906 RunID: run027 16:46:18 Execution failed: Exception in sh: Arguments: [cs.sh] Host: local Directory: mc5-run027/jobs/m/sh-mqnxoyql stderr.txt: stdout.txt: exception @ swift-int.k, line: 511 Caused by: Failed to remove job directory . throw @ swift-int.k, line: 76 k:assign @ swift.k, line: 194 Caused by: Exception in sh: Arguments: [cs.sh] Host: local Directory: mc5-run027/jobs/m/sh-mqnxoyql stderr.txt: stdout.txt: exception @ swift-int.k, line: 511 Caused by: Failed to remove job directory . throw @ swift-int.k, line: 76 swift$ On 3/27/14, 9:11 PM, Mihael Hategan wrote: > I committed the changes. > > This only works for normal swift staging and provider staging. It can be > made to work with wrapper staging, but I have not done so. > > As far as I can tell, there are only two mappers that can map arrays > based on what's on the filesystem rather than what the mapper parameters > are saying: FilesysMapper and SimpleMapper (well, there's also > ConcurrentMapper and FMRIMapper, and, while they should be useable for > collecting output data, it's unlikely that anyone will try to guess how > to name files so that they match what those mappers are expecting). > > To use the feature, do the obvious: > > file[] a ; > // OR > // file[] a ; > // OR > // file[] a ; > > app (file[] oa) gen(int i) { > gen i; // writes a bunch of files of the form foo????.out to > outs/ > } > > a = gen(3); > > I've also committed tests for this. Yadu, if you have some time, can you > check why the check script doesn't find the output from the run? I don't > remember how that worked. > > There are probably going to be a few bugs, since a lot of how the swift > staging data was handled has changed. So please test this as swift in > general. > > One other thing to note is that the globbing is only supported for the > file names. For example: /a/b/??/c/*.txt won't work, but /a/b/x/c/*.txt > will. However, that's if you use a pattern with the FilesysMapper. > Complex structures mapped with SimpleMapper should work. For example, > you should be able to do this: > > type struct { > file a; > file[] b; > } > > app (struct[] o) f(...) {} > > It should figure out the ranges for both o[] and all of o[x].b[]. > > Mihael > > On Wed, 2014-03-26 at 12:04 -0700, Mihael Hategan wrote: >> Fortunately this would not be something that the users would see. >> >> But if they did, a*b -> c*d seems pretty intuitive to me. Maybe even >> more so than >> a(.*)b -> c\1d. It's the implementation of this that is more difficult. >> >> Mihael >> >> On Wed, 2014-03-26 at 13:42 -0500, Michael Wilde wrote: >>> Excellent! >>> >>> The path-difference challenge sounds like the same things we faced in >>> CDM, revisited. >>> >>> I suspect we can find ways to smooth this out in a way that works >>> naturally for users. >>> >>> - Mike >>> >>> >>> On 3/26/14, 1:24 PM, Mihael Hategan wrote: >>>> Almost there. >>>> >>>> It works for non-provider staging. >>>> I'm working on provider staging now. The difficulty is in that providers >>>> must now support glob-pattern staging with a twist. The twist is that >>>> local and remote path names are different. For example, without glob >>>> patterns, a stageout could look like this: >>>> >>>> __root__/dir/a.txt -> /dir/a.txt >>>> >>>> With globs, something like this is possible: >>>> >>>> __root__/dir/a_????_b_????.txt -> /dir/a_????_b_????.txt >>>> >>>> In this case the code needs to recursively glob things and substitute >>>> each glob group in the destination for the respective matching glob in >>>> the source. >>>> >>>> Luckily there are only two providers that support staging at this point: >>>> local and coasters. Unfortunately this has to be implemented twice: once >>>> in Java for local, and once in Perl for coasters. >>>> >>>> Mihael >>>> >>>> On Mon, 2014-03-24 at 08:46 -0700, Mihael Hategan wrote: >>>>> An update on this. >>>>> >>>>> I'm still working on it, but here is the basic idea: >>>>> >>>>> - non-static mappers now support a method that, given a data type, >>>>> returns a list of glob patterns that can be used to search for files >>>>> that could be mapped by that mapper. The list (as opposed to one glob >>>>> pattern) is necessary because there might be cases when you have: >>>>> type s { file a; file[] b}; >>>>> s[] x; >>>>> Then s[] could match either s_????.a or s_????.???? >>>>> - this (possibly empty) list gets sent to _swiftwrap >>>>> - after the job is done, _swiftwrap creates a list of files matching >>>>> those patterns >>>>> - swift-int copies that list back and the files in it and uses the list >>>>> to populate data in a fashion similar to what is done for input >>>>> variables >>>>> >>>>> This is without provider staging. >>>>> >>>>> For provider staging, providers that support staging need to be modified >>>>> to support staging out of files using glob patterns. There might be some >>>>> complications there due to the local vs. remote path naming conventions. >>>>> >>>>> Mihael >>>>> >>>>> On Sat, 2014-03-22 at 14:08 -0500, Michael Wilde wrote: >>>>>> On 3/22/14, 1:38 PM, Mihael Hategan wrote: >>>>>>> My opinion is that this problem is NOT a language/mapper issue. It is an >>>>>>> issue of implementation: how do you get the information about files to a >>>>>>> place where it can be used. >>>>>>> >>>>>>> So I believe that whether we add a new mapper or make it work with >>>>>>> existing mappers, we still need to fix that other more complex problem. >>>>>>> This is the reason why I believe we shouldn't add anything to the >>>>>>> language. >>>>>> Mihael and I discussed this in a chat just now, and I think we are in >>>>>> fact *in* sync. >>>>>> So he's going to push forward on this. >>>>>> >>>>>> - Mike >>>>> _______________________________________________ >>>>> Swift-devel mailing list >>>>> Swift-devel at ci.uchicago.edu >>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >> >> _______________________________________________ >> Swift-devel mailing list >> Swift-devel at ci.uchicago.edu >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > -- Michael Wilde Mathematics and Computer Science Computation Institute Argonne National Laboratory The University of Chicago From wilde at anl.gov Thu May 29 09:08:07 2014 From: wilde at anl.gov (Michael Wilde) Date: Thu, 29 May 2014 09:08:07 -0500 Subject: [Swift-devel] Back to work In-Reply-To: References: Message-ID: <53873F47.8030501@anl.gov> Hi Yadu, welcome back! (at least, part-way back ;) Hope you're having a very pleasant holiday with family and friends. I'll start steering some work your way. First thing, can you: a) check on the automated tests and report to me what they are seeing. What should I and others (Mihael, Justin, Tim) be looking at to see how the tests are doing? b) Get tests in place, and extend the tests, to test the new "array return from app" logic: - test with and without provider staging - test with and without location= (and other stages) - test with a variety of mappers, starting with simple_mapper and array_mapper(s) - test with and without returning non-array files along with array files (in combination with the above...) From the list below, 1-3 is good; please set 4 aside for now. Thanks, - Mike On 5/27/14, 2:41 AM, Yadu Nand wrote: > Hi Mike, > > I've been working a couple of hours spread over weeks on pSIMS for some > additional > changes Joshua wanted. I have the hours noted for those. > > I'll be working in a bit more regular fashion this week, and we can talk if > needed over skype. > > Here's what's on my list: > > 1. Fix the image issue with compute-engine and mail the list. > (My brother is testing this for me) > > 2. Documentation for pSIMS. > > 3. Get the test-suite to report results better. > > 4. Go over the picloud documentation (they've stopped new signups) > > If anything of higher priority comes along, I'll be glad to help. > > ?Thanks,? > > Yadu > -- Michael Wilde Mathematics and Computer Science Computation Institute Argonne National Laboratory The University of Chicago From tim.g.armstrong at gmail.com Thu May 29 09:18:42 2014 From: tim.g.armstrong at gmail.com (Tim Armstrong) Date: Thu, 29 May 2014 09:18:42 -0500 Subject: [Swift-devel] Back to work In-Reply-To: <53873F47.8030501@anl.gov> References: <53873F47.8030501@anl.gov> Message-ID: Welcome back Yadu! This isn't work for Yadu exactly, but it touches on Yadu's coasters work. I was going to flag that I intended to push forward on full integration of asynchronous task executors (GeMTC/Coasters/others) into Swift over the summer, since it's important but I haven't gotten around to adding proper support for multiple worker types, etc: https://code.google.com/p/exm-issues/issues/detail?id=581 https://code.google.com/p/exm-issues/issues/detail?id=503 https://code.google.com/p/exm-issues/issues/detail?id=609 - Tim On Thu, May 29, 2014 at 9:08 AM, Michael Wilde wrote: > Hi Yadu, welcome back! (at least, part-way back ;) > > Hope you're having a very pleasant holiday with family and friends. > > I'll start steering some work your way. First thing, can you: > > a) check on the automated tests and report to me what they are seeing. > What should I and others (Mihael, Justin, Tim) be looking at to see how > the tests are doing? > > b) Get tests in place, and extend the tests, to test the new "array > return from app" logic: > - test with and without provider staging > - test with and without location= (and other stages) > - test with a variety of mappers, starting with simple_mapper and > array_mapper(s) > - test with and without returning non-array files along with array files > (in combination with the above...) > > From the list below, 1-3 is good; please set 4 aside for now. > > Thanks, > > - Mike > > On 5/27/14, 2:41 AM, Yadu Nand wrote: > > Hi Mike, > > > > I've been working a couple of hours spread over weeks on pSIMS for some > > additional > > changes Joshua wanted. I have the hours noted for those. > > > > I'll be working in a bit more regular fashion this week, and we can talk > if > > needed over skype. > > > > Here's what's on my list: > > > > 1. Fix the image issue with compute-engine and mail the list. > > (My brother is testing this for me) > > > > 2. Documentation for pSIMS. > > > > 3. Get the test-suite to report results better. > > > > 4. Go over the picloud documentation (they've stopped new signups) > > > > If anything of higher priority comes along, I'll be glad to help. > > > > ?Thanks,? > > > > Yadu > > > > -- > Michael Wilde > Mathematics and Computer Science Computation Institute > Argonne National Laboratory The University of Chicago > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wilde at anl.gov Thu May 29 09:32:34 2014 From: wilde at anl.gov (Michael Wilde) Date: Thu, 29 May 2014 09:32:34 -0500 Subject: [Swift-devel] Back to work In-Reply-To: References: <53873F47.8030501@anl.gov> Message-ID: <53874502.5070707@anl.gov> Tim, this sounds good. Could you consider how we'd do an async task executor that can use iPython's parallel-remote services, as an alternative to coasters? - Mike ps. Yadu will return to CI on Mon June 2. He's working remotely this week. On 5/29/14, 9:18 AM, Tim Armstrong wrote: > Welcome back Yadu! > > This isn't work for Yadu exactly, but it touches on Yadu's coasters work. > I was going to flag that I intended to push forward on full integration of > asynchronous task executors (GeMTC/Coasters/others) into Swift over the > summer, since it's important but I haven't gotten around to adding proper > support for multiple worker types, etc: > https://code.google.com/p/exm-issues/issues/detail?id=581 > https://code.google.com/p/exm-issues/issues/detail?id=503 > https://code.google.com/p/exm-issues/issues/detail?id=609 > > - Tim > > > On Thu, May 29, 2014 at 9:08 AM, Michael Wilde wrote: > >> Hi Yadu, welcome back! (at least, part-way back ;) >> >> Hope you're having a very pleasant holiday with family and friends. >> >> I'll start steering some work your way. First thing, can you: >> >> a) check on the automated tests and report to me what they are seeing. >> What should I and others (Mihael, Justin, Tim) be looking at to see how >> the tests are doing? >> >> b) Get tests in place, and extend the tests, to test the new "array >> return from app" logic: >> - test with and without provider staging >> - test with and without location= (and other stages) >> - test with a variety of mappers, starting with simple_mapper and >> array_mapper(s) >> - test with and without returning non-array files along with array files >> (in combination with the above...) >> >> From the list below, 1-3 is good; please set 4 aside for now. >> >> Thanks, >> >> - Mike >> >> On 5/27/14, 2:41 AM, Yadu Nand wrote: >>> Hi Mike, >>> >>> I've been working a couple of hours spread over weeks on pSIMS for some >>> additional >>> changes Joshua wanted. I have the hours noted for those. >>> >>> I'll be working in a bit more regular fashion this week, and we can talk >> if >>> needed over skype. >>> >>> Here's what's on my list: >>> >>> 1. Fix the image issue with compute-engine and mail the list. >>> (My brother is testing this for me) >>> >>> 2. Documentation for pSIMS. >>> >>> 3. Get the test-suite to report results better. >>> >>> 4. Go over the picloud documentation (they've stopped new signups) >>> >>> If anything of higher priority comes along, I'll be glad to help. >>> >>> Thanks, >>> >>> Yadu >>> >> -- >> Michael Wilde >> Mathematics and Computer Science Computation Institute >> Argonne National Laboratory The University of Chicago >> >> _______________________________________________ >> Swift-devel mailing list >> Swift-devel at ci.uchicago.edu >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel >> > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel -- Michael Wilde Mathematics and Computer Science Computation Institute Argonne National Laboratory The University of Chicago -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.g.armstrong at gmail.com Thu May 29 10:28:05 2014 From: tim.g.armstrong at gmail.com (Tim Armstrong) Date: Thu, 29 May 2014 10:28:05 -0500 Subject: [Swift-devel] Back to work In-Reply-To: <53874502.5070707@anl.gov> References: <53873F47.8030501@anl.gov> <53874502.5070707@anl.gov> Message-ID: The idea would be to have a generic capability that would let you write C extensions that implement generic executors, plus all of the changes to ADLB/Turbine that let these run alongside CPU workers. Potentially you can add whatever you want. Adding support to the full language stack from syntax down is a separate task. What's the motivation for the IPython idea? It feels like a lot of moving parts to me. - TIm On Thu, May 29, 2014 at 9:32 AM, Michael Wilde wrote: > Tim, this sounds good. Could you consider how we'd do an async task > executor that can use iPython's parallel-remote services, as an alternative > to coasters? > > - Mike > > ps. Yadu will return to CI on Mon June 2. He's working remotely this week. > > > On 5/29/14, 9:18 AM, Tim Armstrong wrote: > > Welcome back Yadu! > > This isn't work for Yadu exactly, but it touches on Yadu's coasters work. > I was going to flag that I intended to push forward on full integration of > asynchronous task executors (GeMTC/Coasters/others) into Swift over the > summer, since it's important but I haven't gotten around to adding proper > support for multiple worker types, etc:https://code.google.com/p/exm-issues/issues/detail?id=581https://code.google.com/p/exm-issues/issues/detail?id=503https://code.google.com/p/exm-issues/issues/detail?id=609 > > - Tim > > > On Thu, May 29, 2014 at 9:08 AM, Michael Wilde wrote: > > > Hi Yadu, welcome back! (at least, part-way back ;) > > Hope you're having a very pleasant holiday with family and friends. > > I'll start steering some work your way. First thing, can you: > > a) check on the automated tests and report to me what they are seeing. > What should I and others (Mihael, Justin, Tim) be looking at to see how > the tests are doing? > > b) Get tests in place, and extend the tests, to test the new "array > return from app" logic: > - test with and without provider staging > - test with and without location= (and other stages) > - test with a variety of mappers, starting with simple_mapper and > array_mapper(s) > - test with and without returning non-array files along with array files > (in combination with the above...) > > From the list below, 1-3 is good; please set 4 aside for now. > > Thanks, > > - Mike > > On 5/27/14, 2:41 AM, Yadu Nand wrote: > > Hi Mike, > > I've been working a couple of hours spread over weeks on pSIMS for some > additional > changes Joshua wanted. I have the hours noted for those. > > I'll be working in a bit more regular fashion this week, and we can talk > > if > > needed over skype. > > Here's what's on my list: > > 1. Fix the image issue with compute-engine and mail the list. > (My brother is testing this for me) > > 2. Documentation for pSIMS. > > 3. Get the test-suite to report results better. > > 4. Go over the picloud documentation (they've stopped new signups) > > If anything of higher priority comes along, I'll be glad to help. > > ?Thanks,? > > Yadu > > > -- > Michael Wilde > Mathematics and Computer Science Computation Institute > Argonne National Laboratory The University of Chicago > > _______________________________________________ > Swift-devel mailing listSwift-devel at ci.uchicago.eduhttps://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > > _______________________________________________ > Swift-devel mailing listSwift-devel at ci.uchicago.eduhttps://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > -- > Michael Wilde > Mathematics and Computer Science Computation Institute > Argonne National Laboratory The University of Chicago > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wilde at anl.gov Thu May 29 10:47:11 2014 From: wilde at anl.gov (Michael Wilde) Date: Thu, 29 May 2014 10:47:11 -0500 Subject: [Swift-devel] iPython parallel coasters In-Reply-To: References: <53873F47.8030501@anl.gov> <53874502.5070707@anl.gov> Message-ID: <5387567F.3070107@anl.gov> [Moving this part of the discussion to a new thread.] Seems like less moving parts than coasters: it would all be in Python (instead of Java + Perl); would have iPython community support; and would be based on a standard protocol (ZeroMQ implementation of AMQP). iPython parallel has almost the exact same process architecture as Swift Coasters. Seemed worth exploring. - Mike On 5/29/14, 10:28 AM, Tim Armstrong wrote: > ... > > What's the motivation for the IPython idea? It feels like a lot of moving > parts to me. > > - TIm > From hategan at mcs.anl.gov Thu May 29 13:47:05 2014 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 29 May 2014 11:47:05 -0700 Subject: [Swift-devel] iPython parallel coasters In-Reply-To: <5387567F.3070107@anl.gov> References: <53873F47.8030501@anl.gov> <53874502.5070707@anl.gov> <5387567F.3070107@anl.gov> Message-ID: <1401389225.27037.6.camel@echo> I was exploring AMQP as an alternative to writing my own protocol (which turned out to be coasters) somewhere before I started working on Swift. In spirit, it seemed to do exactly what I wanted. Unfortunately, there were no reasonable implementations at the time, which I see has changed. So I do think this is worth exploring although I'm not quite sure about the moving parts part. The outsides of things can be deceiving. Mihael On Thu, 2014-05-29 at 10:47 -0500, Michael Wilde wrote: > [Moving this part of the discussion to a new thread.] > > Seems like less moving parts than coasters: it would all be in Python > (instead of Java + Perl); would have iPython community support; and > would be based on a standard protocol (ZeroMQ implementation of AMQP). > > iPython parallel has almost the exact same process architecture as Swift > Coasters. > > Seemed worth exploring. > > - Mike > > On 5/29/14, 10:28 AM, Tim Armstrong wrote: > > ... > > > > What's the motivation for the IPython idea? It feels like a lot of moving > > parts to me. > > > > - TIm > > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel From hategan at mcs.anl.gov Thu May 29 15:21:04 2014 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 29 May 2014 13:21:04 -0700 Subject: [Swift-devel] waiting bugs Message-ID: <1401394864.28239.2.camel@echo> Hi, There is a (relatively low compared to the total) number of bugs that are in waiting state. It might be a good idea to occasionally check and see if there are any bugs that are waiting for YOU to do/say something. Here's a quick link: https://bugzilla.mcs.anl.gov/swift/buglist.cgi?query_format=advanced&bug_status=WAITING+FOR+USER+INPUT&emailreporter1=1&emailtype1=substring&email1=hategan Replace the ending (after the '=' sign) with your email id. Mihael From tim.g.armstrong at gmail.com Thu May 29 15:32:40 2014 From: tim.g.armstrong at gmail.com (Tim Armstrong) Date: Thu, 29 May 2014 15:32:40 -0500 Subject: [Swift-devel] iPython parallel coasters In-Reply-To: <1401389225.27037.6.camel@echo> References: <53873F47.8030501@anl.gov> <53874502.5070707@anl.gov> <5387567F.3070107@anl.gov> <1401389225.27037.6.camel@echo> Message-ID: So you're suggesting to try to speak the IPython protocol and somehow thread task descriptions through that and run the command-line app on the other end? It's partially documented but they seem a little vague about backwards compatibility and whether it's an official public interface. http://ipython.org/ipython-doc/2/development/messaging.html Aside from It seems that it would be difficult (impossible?) to build things like provider staging on top of this. I just tend to think that trying to integrate with existing code causes at least as many problems as it solves unless the other system was really carefully designed to be a building block for other systems. The design seems to assume throughout that your tasks are going to be a Python function. If anything we might want something that provides lower-level functionality that isn't specialised for running Python functions. - Tim On Thu, May 29, 2014 at 1:47 PM, Mihael Hategan wrote: > I was exploring AMQP as an alternative to writing my own protocol (which > turned out to be coasters) somewhere before I started working on Swift. > In spirit, it seemed to do exactly what I wanted. Unfortunately, there > were no reasonable implementations at the time, which I see has changed. > > So I do think this is worth exploring although I'm not quite sure about > the moving parts part. The outsides of things can be deceiving. > > Mihael > > On Thu, 2014-05-29 at 10:47 -0500, Michael Wilde wrote: > > [Moving this part of the discussion to a new thread.] > > > > Seems like less moving parts than coasters: it would all be in Python > > (instead of Java + Perl); would have iPython community support; and > > would be based on a standard protocol (ZeroMQ implementation of AMQP). > > > > iPython parallel has almost the exact same process architecture as Swift > > Coasters. > > > > Seemed worth exploring. > > > > - Mike > > > > On 5/29/14, 10:28 AM, Tim Armstrong wrote: > > > ... > > > > > > What's the motivation for the IPython idea? It feels like a lot of > moving > > > parts to me. > > > > > > - TIm > > > > > > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wilde at anl.gov Thu May 29 15:43:18 2014 From: wilde at anl.gov (Michael Wilde) Date: Thu, 29 May 2014 15:43:18 -0500 Subject: [Swift-devel] iPython parallel coasters In-Reply-To: References: <53873F47.8030501@anl.gov> <53874502.5070707@anl.gov> <5387567F.3070107@anl.gov> <1401389225.27037.6.camel@echo> Message-ID: <53879BE6.9040009@anl.gov> On 5/29/14, 3:32 PM, Tim Armstrong wrote: > Aside from It seems that it would be difficult (impossible?) to build > things like provider staging on top of this. I'd envision using GridFTP for staging; it seems to be well-callable from Python, as demonstrated by the GO transfer client (as well as, I think, SAGA). Having said all that - and started this thread - I'd also like to see the coaster binding to Swift/T actually work to the point of user-readiness. I've got to believe, Mihael, Justin, Tim, and Yadu, that if you set your mind to this and worked closely on it, you could jointly break through all current problems and actually make this work, provider-staging included. - Mike From tim.g.armstrong at gmail.com Thu May 29 15:56:00 2014 From: tim.g.armstrong at gmail.com (Tim Armstrong) Date: Thu, 29 May 2014 15:56:00 -0500 Subject: [Swift-devel] iPython parallel coasters In-Reply-To: <53879BE6.9040009@anl.gov> References: <53873F47.8030501@anl.gov> <53874502.5070707@anl.gov> <5387567F.3070107@anl.gov> <1401389225.27037.6.camel@echo> <53879BE6.9040009@anl.gov> Message-ID: Sure, I just think the problem is that it hasn't been an urgent priority for anyone. I don't think there's a blocking problem it's just a bunch of integration and testing work to get it going well. I was planning to do the bits I need to do over the summer and start bugging people about it as well. On Thu, May 29, 2014 at 3:43 PM, Michael Wilde wrote: > > On 5/29/14, 3:32 PM, Tim Armstrong wrote: > >> Aside from It seems that it would be difficult (impossible?) to build >> things like provider staging on top of this. >> > I'd envision using GridFTP for staging; it seems to be well-callable from > Python, as demonstrated by the GO transfer client (as well as, I think, > SAGA). > > Having said all that - and started this thread - I'd also like to see the > coaster binding to Swift/T actually work to the point of user-readiness. > > I've got to believe, Mihael, Justin, Tim, and Yadu, that if you set your > mind to this and worked closely on it, you could jointly break through all > current problems and actually make this work, provider-staging included. > > - Mike > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wilde at anl.gov Fri May 30 08:40:29 2014 From: wilde at anl.gov (Michael Wilde) Date: Fri, 30 May 2014 08:40:29 -0500 Subject: [Swift-devel] 0.95 release and link In-Reply-To: <5387F315.5030302@anl.gov> References: <5387F315.5030302@anl.gov> Message-ID: <53888A4D.60303@anl.gov> David, at the start of the trunk User Guide, this link is bad: * Download Swift 0.95 athttp://swiftlang.org/packages/swift-0.95.tar.gz. Whats the latest 0.95? Are we ready to call the latest release candidate the final 0.95 release? - Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From wilde at anl.gov Fri May 30 17:16:02 2014 From: wilde at anl.gov (Michael Wilde) Date: Fri, 30 May 2014 17:16:02 -0500 Subject: [Swift-devel] web site changes in progress Message-ID: <53890322.5090404@anl.gov> Ive removed/disabled the old main/ directory and the forwarding link, with direct edits to the swift/ www dir. I'll get these into svn shortly and try to locate the automated update mechanism. Then I'll create a swift-t directory for you, Justin. - Mike -- Michael Wilde Mathematics and Computer Science Computation Institute Argonne National Laboratory The University of Chicago From wilde at anl.gov Fri May 30 20:23:58 2014 From: wilde at anl.gov (Michael Wilde) Date: Fri, 30 May 2014 20:23:58 -0500 Subject: [Swift-devel] web site changes in progress In-Reply-To: <53890322.5090404@anl.gov> References: <53890322.5090404@anl.gov> Message-ID: <53892F2E.7080803@anl.gov> Justin: David added a brief swift-devel doc about how to make changes, at https://sites.google.com/site/swiftdevel/home/updating-web You have several files in the swift www dir that are not group-writeable. WHen you get a chance, can you fix those so that we can do clea pushes of web content? All: I un-did my changes to eliminate the main/ dir, and instead eliminated just the redirect message. I could not easily get my prior changes to work in both the live site (under the document root) and on a test site (under public_html/www). As I recall now, that was the reason for the redirect originally. Thanks, - Mike On 5/30/14, 5:16 PM, Michael Wilde wrote: > Ive removed/disabled the old main/ directory and the forwarding link, > with direct edits to the swift/ www dir. > > I'll get these into svn shortly and try to locate the automated update > mechanism. > > Then I'll create a swift-t directory for you, Justin. > > - Mike > -- Michael Wilde Mathematics and Computer Science Computation Institute Argonne National Laboratory The University of Chicago From wilde at anl.gov Sat May 31 14:46:43 2014 From: wilde at anl.gov (Michael Wilde) Date: Sat, 31 May 2014 14:46:43 -0500 Subject: [Swift-devel] Swift web changes pending Message-ID: <538A31A3.9030007@anl.gov> Hi All, I added TrySwift and Swift/T to the Swift main page. You can preview the changes at: http://web.ci.uchicago.edu/~wilde/www/main/ The changes are committed to svn but not yet pushed to the main site. I added the GeMTC paper under "Whats New" after Swift/T. I created a new main directory Swift-T/ for Swift/T. At the moment, this just forwards to the Google exm site Swift-T page. Justin: feel free to start integrating Swift/T content below this directory. You can check out the entire web below your public_html directory, test there, and commit. After youre done and committed, I'll update my test copy, check it out, and then push to the live site some time tomorrow. Im going to shift later to work on TrySwift text (probably not till tomorrow morning). Yadu is looking at creating a Local Host tutorial version that runs on Linux; hopefully same will run on Mac. Justin, Tim: do you know how to create a nice mac install package? Did you do so for Swift/T? Thanks, - Mike -- Michael Wilde Mathematics and Computer Science Computation Institute Argonne National Laboratory The University of Chicago From wilde at anl.gov Sat May 31 14:51:17 2014 From: wilde at anl.gov (Michael Wilde) Date: Sat, 31 May 2014 14:51:17 -0500 Subject: [Swift-devel] Swift web changes pending In-Reply-To: <538A31A3.9030007@anl.gov> References: <538A31A3.9030007@anl.gov> Message-ID: <538A32B5.2040708@anl.gov> Hi Vytas, can you suggest how to shrink the extra white space between the rotator and the grid cels below it? (in http://web.ci.uchicago.edu/~wilde/www ) I assume these are pushed down because the right col grew. Can we re-orient the colums so that the right col goes top-to-bottom, I assume? Suggestions/improvements welcome. If you have time, feel free to commit to svn and I will test locally. Also note: when testing locally, the downloads dont work unless you copy in the packages/ dir, and tryswift/ doesnt work unless you install it from SwiftApps in svn. Thanks! - Mike On 5/31/14, 2:46 PM, Michael Wilde wrote: > Hi All, > > I added TrySwift and Swift/T to the Swift main page. You can preview > the changes at: > > http://web.ci.uchicago.edu/~wilde/www/main/ > > The changes are committed to svn but not yet pushed to the main site. > > I added the GeMTC paper under "Whats New" after Swift/T. > > I created a new main directory Swift-T/ for Swift/T. At the moment, > this just forwards to the Google exm site Swift-T page. > > Justin: feel free to start integrating Swift/T content below this directory. > > You can check out the entire web below your public_html directory, test > there, and commit. > > After youre done and committed, I'll update my test copy, check it out, > and then push to the live site some time tomorrow. > > Im going to shift later to work on TrySwift text (probably not till > tomorrow morning). > > Yadu is looking at creating a Local Host tutorial version that runs on > Linux; hopefully same will run on Mac. > > Justin, Tim: do you know how to create a nice mac install package? Did > you do so for Swift/T? > > Thanks, > > - Mike > -- Michael Wilde Mathematics and Computer Science Computation Institute Argonne National Laboratory The University of Chicago