From wilde at mcs.anl.gov Mon Oct 8 09:40:12 2012 From: wilde at mcs.anl.gov (Michael Wilde) Date: Mon, 8 Oct 2012 09:40:12 -0500 Subject: [Swift-user] CFP: 2nd IEEE International Workshop on Workflow Models, Systems, Services and Applications in the Cloud (CloudFlow) 2013 In-Reply-To: References: Message-ID: * * *Second IEEE International Workshop on Workflow Models, Systems, Services and Applications in the Cloud (CloudFlow) 2013* *To be held in conjunction with the 27th IEEE International Parallel & Distributed Processing Symposium (IPDPS) 2013, Cambridge, Boston, Massachusetts, USA, May 20-24, 2013.* http://www.cloud-uestc.cn/cloudflow/home.html *Overview* Cloud computing is gaining tremendous momentum in both academia and industry, more and more people are migrating their data and applications into the Cloud. We have observed wide adoption of the MapReduce computing model and the open source Hadoop system for large scale distributed data processing, and a variety of ad hoc mashup techniques that weave together Web applications. However, these are just first steps towards managing complex task and data dependencies in the Cloud, as there are more challenging issues such as large parameter space exploration, data partitioning and distribution, scheduling and optimization, smart reruns, and provenance tracking associated with workflow execution. Cloud needs structured and mature workflow technologies to handle such issues, and vice versa, as Cloud offers unprecedented scalability to workflow systems, and could potentially change the way we perceive and conduct research and experiments. The scale and complexity of the science and data analytics problems that can be handled can be greatly increased on the Cloud, and the on-demand nature of resource allocation on the Cloud will also help improve resource utilization and user experience. As Cloud computing provides a paradigm-shifting utility-oriented computing model in terms of the unprecedented size of datacenter-level resource pool and the on-demand resource provisioning mechanism, there are lots of challenges in bringing Cloud and workflows together. We need high level languages and computing models for large scale workflow specification; we need to adapt existing workflow architectures into the Cloud, and integrate workflow systems with Cloud infrastructure and resources; we also need to leverage Cloud data storage technologies to efficiently distribute data over a large number of nodes and explore data locality during computation etc. We organize the CloudFlow workshop as a venue for the workflow and Cloud communities to define models and paradigms, present their state-of-the-art work, share their thoughts and experiences, and explore new directions in realizing workflows in the Cloud. *Topics:* We welcome the submission of original work related to the topics listed below, which include (in the context of Cloud): ? Models and Languages for Large Scale Workflow Specification ? Workflow Architecture and Framework ? Large Scale Workflow Systems ? Service Workflow ? Workflow Composition and Orchestration ? Workflow Migration into the Cloud ? Workflow Scheduling and Optimization ? Cloud Middleware in Support of Workflow ? Virtualized Environment ? Workflow Applications and Case Studies ? Performance and Scalability Analysis ? Peta-Scale Data Processing ? Event Processing and Messaging ? Real-Time Analytics ? Provenance *Paper Submission* Authors are invited to submit papers with unpublished, original work. The papers should not exceed 10 single-spaced double-column pages using 10-point size font on 8.5x11 inch pages (IEEE conference style), including figures, tables, and references. Paper submission should be done via the online CMT system, Microsoft?s Academic Conference Management Service (* https://cmt.research.microsoft.com/CF2013*) by midnight January 9th, 2013 Pacific Time. The final format should be in PDF. Proceedings of the workshop will be published by the IEEE Digital Library (indexed by EI) and distributed at the conference. Selected excellent work may be eligible for additional post-conference publication as journal articles or book chapters. Submission implies the willingness of at least one of the authors to register and present the paper. *Important Dates* ** Paper submission: January 9th, 2013 Acceptance notification: February 8th, 2013 Final paper due: Feb 19th, 2013 *Organization* Workshop Chairs: Dr. Yong Zhao University of Electronic Science and Technology of China, China yongzh04 at gmail.com Dr. Cui Lin California State University, Fresno, USA clin at csufresno.edu Dr. Shiyong Lu Wayne State University, USA shiyong at wayne.edu Program Chair: Dr. Wenhong Tian University of Electronic Science and Technology of China, China Publicity Chair: Dr. Ruini Xue University of Electronic Science and Technology of China, China *Steering Committee * ? Daniel S. Katz, University of Chicago, U.S.A. ? Mike Wilde, University of Chicago, U.S.A. ? Ewa Deelman, University of South California, U.S.A. ? Tevfik Kosar, University at Buffalo, U.S.A. ? Ilkay Altintas, San Diego Supercomputer Center, U.S.A. ? Ioan Raicu, Illinois Institute of Technology, U.S.A. ? Yogesh Simmhan, University of Southern California, U.S.A. ? Ian Taylor, Cardiff University, U.K. ? Weimin Zheng, Tsinghua University, China ? Hai Jin, Huazhong University of Science and Engineering, China ? Wanchun Dou, Nanjing University, China ? Hui Zhang, National Science and Technology Infrastructure, China *Program Committee * ? Shawn Bowers, Gonzaga University, U.S.A. ? Douglas Thain, University of Notre Dame, U.S.A. ? Ian Gorton, Pacific Northwest National Laboratory, U.S.A. ? Artem Chebotko, University of Texas at Pan American, U.S.A. ? Weisong Shi, Wayne State University, U.S.A. ? Paolo Missier, Newcastle University, U.K. ? Wei Tan, IBM T. J. Watson Research Center, U.S.A. ? Jianwu Wang, San Diego Super Computer Center, U.S.A. ? Ping Yang, Binghamton University, U.S.A. ? Jian Guo, Harvard University, U.S.A. ? Liqiang Wang, University of Wyoming, U.S.A. ? Paul Groth, VU University Amsterdam, the Netherlands ? Zhiming Zhao, University of Amsterdam, the Netherlands ? Marta Mattoso, Federal University of Rio de Janeiro, Brazil ? Wenhong Tian, University of Electronic Science and Technology of China, China ? Ruini Xue, Tsinghua University, China ? Jian Cao, Shanghai Jiaotong University, China ? Jianxun Liu, Hunan University of Science and Technology, China ? Song Zhang, Chinese Academy of Sciences, China ? Hua Hu, Hangzhou Dianzi University, China -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmargolpeople at gmail.com Mon Oct 8 14:29:33 2012 From: jmargolpeople at gmail.com (Jonathan Margoliash) Date: Mon, 8 Oct 2012 14:29:33 -0500 Subject: [Swift-user] A question about design choice Message-ID: Hello swift-user, I had a question about the design of my program. My program involves a fair bit of matrix manipulations, so instead of writing the control structure of the program in swift, I've written most of the logic and data management in matlab scripts. The only functionality of swift that I use is to parallelize large for-loops, while the code that runs my swift scripts, and the apps that swift distributes inside each for-loop, are written in matlab. In essence, I've created a program in matlab which calls a swift script many times in serial, using swift to distributing many jobs in parallel. The issue I've come across is that after a certain number of serial calls to swift, I need to do a significant amount of data crunching before I can resume making serial swift calls. This data crunching is resource-intensive enough that it can't be done on a log in node, but not nearly intensive enough that it needs to be broken down into component parts and parallelized. My question is, what is the correct way of handling this situation? Does it make sense to write a separate swift script whose body only contains a single app call, so I can get this minor job run on one of the compute nodes? If I do it that way, will there likely be a significant delay while the scheduler processes my request, or is it likely to go through quickly because I'm only asking for one node for a short period of time? Thanks for your comments, Jonathan -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Mon Oct 8 15:31:47 2012 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 08 Oct 2012 13:31:47 -0700 Subject: [Swift-user] A question about design choice In-Reply-To: References: Message-ID: <1349728307.10418.0.camel@blabla> The swift script you have for the parallel jobs, is it a simple foreach loop? Mihael On Mon, 2012-10-08 at 14:29 -0500, Jonathan Margoliash wrote: > Hello swift-user, > > > I had a question about the design of my program. My program involves a > fair bit of matrix manipulations, so instead of writing the control > structure of the program in swift, I've written most of the logic and > data management in matlab scripts. The only functionality of swift > that I use is to parallelize large for-loops, while the code that runs > my swift scripts, and the apps that swift distributes inside each > for-loop, are written in matlab. In essence, I've created a program in > matlab which calls a swift script many times in serial, using swift to > distributing many jobs in parallel. > > > The issue I've come across is that after a certain number of serial > calls to swift, I need to do a significant amount of data crunching > before I can resume making serial swift calls. This data crunching is > resource-intensive enough that it can't be done on a log in node, but > not nearly intensive enough that it needs to be broken down into > component parts and parallelized. My question is, what is the correct > way of handling this situation? Does it make sense to write a separate > swift script whose body only contains a single app call, so I can get > this minor job run on one of the compute nodes? If I do it that way, > will there likely be a significant delay while the scheduler processes > my request, or is it likely to go through quickly because I'm only > asking for one node for a short period of time? > > > Thanks for your comments, > > > Jonathan > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user From jmargolpeople at gmail.com Mon Oct 8 15:35:49 2012 From: jmargolpeople at gmail.com (Jonathan Margoliash) Date: Mon, 8 Oct 2012 15:35:49 -0500 Subject: [Swift-user] A question about design choice In-Reply-To: <1349728307.10418.0.camel@blabla> References: <1349728307.10418.0.camel@blabla> Message-ID: Along with a couple typedefs and an app definition, yes. Is there a better way to do what I'm doing? Thanks, Jonathan On Mon, Oct 8, 2012 at 3:31 PM, Mihael Hategan wrote: > The swift script you have for the parallel jobs, is it a simple foreach > loop? > > Mihael > > On Mon, 2012-10-08 at 14:29 -0500, Jonathan Margoliash wrote: > > Hello swift-user, > > > > > > I had a question about the design of my program. My program involves a > > fair bit of matrix manipulations, so instead of writing the control > > structure of the program in swift, I've written most of the logic and > > data management in matlab scripts. The only functionality of swift > > that I use is to parallelize large for-loops, while the code that runs > > my swift scripts, and the apps that swift distributes inside each > > for-loop, are written in matlab. In essence, I've created a program in > > matlab which calls a swift script many times in serial, using swift to > > distributing many jobs in parallel. > > > > > > The issue I've come across is that after a certain number of serial > > calls to swift, I need to do a significant amount of data crunching > > before I can resume making serial swift calls. This data crunching is > > resource-intensive enough that it can't be done on a log in node, but > > not nearly intensive enough that it needs to be broken down into > > component parts and parallelized. My question is, what is the correct > > way of handling this situation? Does it make sense to write a separate > > swift script whose body only contains a single app call, so I can get > > this minor job run on one of the compute nodes? If I do it that way, > > will there likely be a significant delay while the scheduler processes > > my request, or is it likely to go through quickly because I'm only > > asking for one node for a short period of time? > > > > > > Thanks for your comments, > > > > > > Jonathan > > _______________________________________________ > > Swift-user mailing list > > Swift-user at ci.uchicago.edu > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Mon Oct 8 15:40:21 2012 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 08 Oct 2012 13:40:21 -0700 Subject: [Swift-user] A question about design choice In-Reply-To: References: <1349728307.10418.0.camel@blabla> Message-ID: <1349728821.10963.0.camel@blabla> On Mon, 2012-10-08 at 15:35 -0500, Jonathan Margoliash wrote: > Along with a couple typedefs and an app definition, yes. Is there a > better way to do what I'm doing? Thanks, Are you using a single cluster or are you planning to branch out to multiple clusters? From jmargolpeople at gmail.com Mon Oct 8 15:45:45 2012 From: jmargolpeople at gmail.com (Jonathan Margoliash) Date: Mon, 8 Oct 2012 15:45:45 -0500 Subject: [Swift-user] A question about design choice In-Reply-To: <1349728821.10963.0.camel@blabla> References: <1349728307.10418.0.camel@blabla> <1349728821.10963.0.camel@blabla> Message-ID: In my previous email I forgot to mention that I'm using a doubly-nested for loop, but I doubt that matters. Currently we are using a single cluster. I'm not sure if there are plans to expand to using multiple clusters at once. On Mon, Oct 8, 2012 at 3:40 PM, Mihael Hategan wrote: > On Mon, 2012-10-08 at 15:35 -0500, Jonathan Margoliash wrote: > > Along with a couple typedefs and an app definition, yes. Is there a > > better way to do what I'm doing? Thanks, > > Are you using a single cluster or are you planning to branch out to > multiple clusters? > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Mon Oct 8 16:12:47 2012 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 08 Oct 2012 14:12:47 -0700 Subject: [Swift-user] A question about design choice In-Reply-To: References: <1349728307.10418.0.camel@blabla> <1349728821.10963.0.camel@blabla> Message-ID: <1349730767.10963.16.camel@blabla> We wrote swift to automatically sort out complex parallelism between bunches of heavy work. You seem to be doing that in Matlab. Assuming that it's not trivial or advantageous to switch to doing those in swift, it looks like you need a mechanism to efficiently deal with those bunches of heavy work. I think the quickest path for you would be to use a persistent coaster service. You would be able to call swift as you do now, but different swift runs would re-use the same pbs jobs to run your apps. If you think that swift is unnecessary (and a parallel for loop is something for which I'd say swift is only marginally useful), there might be some compromise solutions, but they may require a bit more work for benefits that I cannot quantify well, but I suspect to not be significant. So I think that if you want to explore the latter, you should probably think in terms of specific and tangible downsides with the former. So please list those, and we can start from there if needed. Mihael On Mon, 2012-10-08 at 15:45 -0500, Jonathan Margoliash wrote: > In my previous email I forgot to mention that I'm using a > doubly-nested for loop, but I doubt that matters. > > > Currently we are using a single cluster. I'm not sure if there are > plans to expand to using multiple clusters at once. > > On Mon, Oct 8, 2012 at 3:40 PM, Mihael Hategan > wrote: > On Mon, 2012-10-08 at 15:35 -0500, Jonathan Margoliash wrote: > > Along with a couple typedefs and an app definition, yes. Is > there a > > better way to do what I'm doing? Thanks, > > > Are you using a single cluster or are you planning to branch > out to > multiple clusters? > > > From isadoogh at iit.edu Tue Oct 23 17:46:57 2012 From: isadoogh at iit.edu (Iman Sadooghi) Date: Tue, 23 Oct 2012 17:46:57 -0500 Subject: [Swift-user] running swift k on multiple instances on EC2, using coaster-service Message-ID: Hi everyone I am trying to run a Montage application workflow with swift on multiple instances of AMAZON EC2. So far I was able to set up a cluster, and a PVFS files system shared among the nodes ( using FUSE. so I will have POSIX interface on my *swift work directory*). I have tried running a simple hello.swift example on multiple nodes with the coaster. the working directory is the shared folder (supported by PVFS). when I run the code using my own tc.data and sites.xml, this will happen: (my command) ubuntu at ip-10-244-4-101:~/coaster$ swift -tc.file tc.data -sites.file sites.xml ~/swift-0.93/examples/swift/tutorial/hello.swift (results:) Swift 0.93 swift-r5483 cog-r3339 RunID: 20121023-2200-4d3knr72 Progress: time: Tue, 23 Oct 2012 22:00:50 +0000 Find: http://10.244.4.101:1213 Find: keepalive(120), reconnect - http://10.244.4.101:1213 Passive queue processor initialized. Callback URI is http://10.244.4.101:1212 Progress: time: Tue, 23 Oct 2012 22:01:20 +0000 Submitted:1 Progress: time: Tue, 23 Oct 2012 22:01:50 +0000 Submitted:1 Progress: time: Tue, 23 Oct 2012 22:02:20 +0000 Submitted:1 Progress: time: Tue, 23 Oct 2012 22:02:50 +0000 Submitted:1 Progress: time: Tue, 23 Oct 2012 22:03:20 +0000 Submitted:1 Progress: time: Tue, 23 Oct 2012 22:03:50 +0000 Submitted:1 Progress: time: Tue, 23 Oct 2012 22:04:20 +0000 Submitted:1 and it keeps doing this forever meaning that there is no answer from worker nodes! as I checked on worker nodes, the working files are created on the shared folder, and when i check the running applications, there is a java application running. but nothing happens. I have also attached the log file of my hello.swift running in case you need to take a look at it. should I consider using pbs, or condor,... I have no idea about how they work though. I appreciate if anyone can help me with it. Thank you so much. Best, -- Iman Sadooghi Illinois Institute of Technology (IIT) Data-Intensive Distributed Systems Laboratory -------------- next part -------------- An HTML attachment was scrubbed... URL: From ketancmaheshwari at gmail.com Tue Oct 23 18:01:15 2012 From: ketancmaheshwari at gmail.com (Ketan Maheshwari) Date: Tue, 23 Oct 2012 19:01:15 -0400 Subject: [Swift-user] running swift k on multiple instances on EC2, using coaster-service In-Reply-To: References: Message-ID: Hi Iman, On the worker nodes, do you see worker.pl running? That must be running for any work to happen on those nodes. Another possibility is that the workers on nodes are not seeing the service running on 10.x.y.z ip. If the service is running on an EC2 node, you will see another ip which you might try by putting in your sites file service url. On Tue, Oct 23, 2012 at 6:46 PM, Iman Sadooghi wrote: > Hi everyone > > I am trying to run a Montage application workflow with swift on multiple > instances of AMAZON EC2. > So far I was able to set up a cluster, and a PVFS files system shared > among the nodes ( using FUSE. so I will have POSIX interface on my *swift > work directory*). > I have tried running a simple hello.swift example on multiple nodes with > the coaster. the working directory is the shared folder (supported by PVFS). > when I run the code using my own tc.data and sites.xml, this will happen: > > (my command) ubuntu at ip-10-244-4-101:~/coaster$ swift -tc.file tc.data > -sites.file sites.xml ~/swift-0.93/examples/swift/tutorial/hello.swift > (results:) > Swift 0.93 swift-r5483 cog-r3339 > > RunID: 20121023-2200-4d3knr72 > Progress: time: Tue, 23 Oct 2012 22:00:50 +0000 > Find: http://10.244.4.101:1213 > Find: keepalive(120), reconnect - http://10.244.4.101:1213 > Passive queue processor initialized. Callback URI is > http://10.244.4.101:1212 > Progress: time: Tue, 23 Oct 2012 22:01:20 +0000 Submitted:1 > Progress: time: Tue, 23 Oct 2012 22:01:50 +0000 Submitted:1 > Progress: time: Tue, 23 Oct 2012 22:02:20 +0000 Submitted:1 > Progress: time: Tue, 23 Oct 2012 22:02:50 +0000 Submitted:1 > Progress: time: Tue, 23 Oct 2012 22:03:20 +0000 Submitted:1 > Progress: time: Tue, 23 Oct 2012 22:03:50 +0000 Submitted:1 > Progress: time: Tue, 23 Oct 2012 22:04:20 +0000 Submitted:1 > > and it keeps doing this forever meaning that there is no answer from > worker nodes! > as I checked on worker nodes, the working files are created on the shared > folder, and when i check the running applications, there is a java > application running. but nothing happens. > I have also attached the log file of my hello.swift running in case you > need to take a look at it. > should I consider using pbs, or condor,... I have no idea about how they > work though. > > I appreciate if anyone can help me with it. Thank you so much. > > Best, > -- > Iman Sadooghi > Illinois Institute of Technology (IIT) > Data-Intensive Distributed Systems Laboratory > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > -- Ketan -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidk at ci.uchicago.edu Wed Oct 24 00:47:01 2012 From: davidk at ci.uchicago.edu (David Kelly) Date: Wed, 24 Oct 2012 00:47:01 -0500 (CDT) Subject: [Swift-user] running swift k on multiple instances on EC2, using coaster-service In-Reply-To: Message-ID: <84123551.161065.1351057621976.JavaMail.root@zimbra-mb2.anl.gov> Hello Iman, Could you please explain how you are trying to set up coasters? Are you manually running worker.pl on each node and pointing it to the address/port, or using a script like start-coaster-service? Is coaster-service running on the same machine as swift? Are you setting up the instances with any firewall rules in place? Could you please send the sites.xml you're using, and the log file from hello.swift (or a link to it, if it's too large to attach?) Thanks! Regards, David ----- Original Message ----- > From: "Iman Sadooghi" > To: swift-user at ci.uchicago.edu > Sent: Tuesday, October 23, 2012 5:46:57 PM > Subject: [Swift-user] running swift k on multiple instances on EC2, using coaster-service > Hi everyone > > > I am trying to run a Montage application workflow with swift on > multiple instances of AMAZON EC2. > So far I was able to set up a cluster, and a PVFS files system shared > among the nodes ( using FUSE. so I will have POSIX interface on my > swift work directory ). > I have tried running a simple hello.swift example on multiple nodes > with the coaster. the working directory is the shared folder > (supported by PVFS). > when I run the code using my own tc.data and sites.xml, this will > happen: > > > > (my command) ubuntu at ip-10-244-4-101:~/coaster$ swift -tc.file tc.data > -sites.file sites.xml ~/swift-0.93/examples/swift/tutorial/hello.swift > (results:) > Swift 0.93 swift-r5483 cog-r3339 > > > RunID: 20121023-2200-4d3knr72 > Progress: time: Tue, 23 Oct 2012 22:00:50 +0000 > Find: http://10.244.4.101:1213 > Find: keepalive(120), reconnect - http://10.244.4.101:1213 > Passive queue processor initialized. Callback URI is > http://10.244.4.101:1212 > Progress: time: Tue, 23 Oct 2012 22:01:20 +0000 Submitted:1 > Progress: time: Tue, 23 Oct 2012 22:01:50 +0000 Submitted:1 > Progress: time: Tue, 23 Oct 2012 22:02:20 +0000 Submitted:1 > Progress: time: Tue, 23 Oct 2012 22:02:50 +0000 Submitted:1 > Progress: time: Tue, 23 Oct 2012 22:03:20 +0000 Submitted:1 > Progress: time: Tue, 23 Oct 2012 22:03:50 +0000 Submitted:1 > Progress: time: Tue, 23 Oct 2012 22:04:20 +0000 Submitted:1 > > > and it keeps doing this forever meaning that there is no answer from > worker nodes! > as I checked on worker nodes, the working files are created on the > shared folder, and when i check the running applications, there is a > java application running. but nothing happens. > I have also attached the log file of my hello.swift running in case > you need to take a look at it. > should I consider using pbs, or condor,... I have no idea about how > they work though. > > > I appreciate if anyone can help me with it. Thank you so much. > > Best, -- > Iman Sadooghi > Illinois Institute of Technology (IIT) > Data-Intensive Distributed Systems Laboratory > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user From davidk at ci.uchicago.edu Wed Oct 24 10:28:50 2012 From: davidk at ci.uchicago.edu (David Kelly) Date: Wed, 24 Oct 2012 10:28:50 -0500 (CDT) Subject: [Swift-user] running swift k on multiple instances on EC2, using coaster-service In-Reply-To: Message-ID: <1925150560.162848.1351092530047.JavaMail.root@zimbra-mb2.anl.gov> Iman, There is some documentation on how to use the start-coaster-service script at http://www.ci.uchicago.edu/swift/guides/release-0.93/siteguide/siteguide.html#_bag_of_workstations. You should be able to follow the "bag of workstations" configuration example once you have started your EC2 instance. That may help to simplify things for you a bit. Please let me know if you have any questions or issues with this. Thanks, David ----- Original Message ----- > From: "Iman Sadooghi" > To: swift-user at ci.uchicago.edu > Sent: Tuesday, October 23, 2012 5:46:57 PM > Subject: [Swift-user] running swift k on multiple instances on EC2, using coaster-service > Hi everyone > > > I am trying to run a Montage application workflow with swift on > multiple instances of AMAZON EC2. > So far I was able to set up a cluster, and a PVFS files system shared > among the nodes ( using FUSE. so I will have POSIX interface on my > swift work directory ). > I have tried running a simple hello.swift example on multiple nodes > with the coaster. the working directory is the shared folder > (supported by PVFS). > when I run the code using my own tc.data and sites.xml, this will > happen: > > > > (my command) ubuntu at ip-10-244-4-101:~/coaster$ swift -tc.file tc.data > -sites.file sites.xml ~/swift-0.93/examples/swift/tutorial/hello.swift > (results:) > Swift 0.93 swift-r5483 cog-r3339 > > > RunID: 20121023-2200-4d3knr72 > Progress: time: Tue, 23 Oct 2012 22:00:50 +0000 > Find: http://10.244.4.101:1213 > Find: keepalive(120), reconnect - http://10.244.4.101:1213 > Passive queue processor initialized. Callback URI is > http://10.244.4.101:1212 > Progress: time: Tue, 23 Oct 2012 22:01:20 +0000 Submitted:1 > Progress: time: Tue, 23 Oct 2012 22:01:50 +0000 Submitted:1 > Progress: time: Tue, 23 Oct 2012 22:02:20 +0000 Submitted:1 > Progress: time: Tue, 23 Oct 2012 22:02:50 +0000 Submitted:1 > Progress: time: Tue, 23 Oct 2012 22:03:20 +0000 Submitted:1 > Progress: time: Tue, 23 Oct 2012 22:03:50 +0000 Submitted:1 > Progress: time: Tue, 23 Oct 2012 22:04:20 +0000 Submitted:1 > > > and it keeps doing this forever meaning that there is no answer from > worker nodes! > as I checked on worker nodes, the working files are created on the > shared folder, and when i check the running applications, there is a > java application running. but nothing happens. > I have also attached the log file of my hello.swift running in case > you need to take a look at it. > should I consider using pbs, or condor,... I have no idea about how > they work though. > > > I appreciate if anyone can help me with it. Thank you so much. > > Best, -- > Iman Sadooghi > Illinois Institute of Technology (IIT) > Data-Intensive Distributed Systems Laboratory > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user From iraicu at cs.iit.edu Wed Oct 24 11:47:17 2012 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Wed, 24 Oct 2012 11:47:17 -0500 Subject: [Swift-user] running swift k on multiple instances on EC2, using coaster-service In-Reply-To: <1925150560.162848.1351092530047.JavaMail.root@zimbra-mb2.anl.gov> References: <1925150560.162848.1351092530047.JavaMail.root@zimbra-mb2.anl.gov> Message-ID: <50881B95.6070202@cs.iit.edu> Hi David, Has anyone in the Swift group run Swift on Amazon? Perhaps there are already some pre-configured images Iman can use, that already has the environment (Linux, Java, Swift, PBS/Condor, GridFTP, NFS/PVFS, etc) setup and ready to use. If there are such images, please point Iman in the right direction. If not, perhaps after Iman finishes his configuration, we can reference his images for others to use on Amazon. Ioan On 10/24/2012 10:28 AM, David Kelly wrote: > Iman, > > There is some documentation on how to use the start-coaster-service script at http://www.ci.uchicago.edu/swift/guides/release-0.93/siteguide/siteguide.html#_bag_of_workstations. You should be able to follow the "bag of workstations" configuration example once you have started your EC2 instance. That may help to simplify things for you a bit. Please let me know if you have any questions or issues with this. > > Thanks, > David > > ----- Original Message ----- >> From: "Iman Sadooghi" >> To: swift-user at ci.uchicago.edu >> Sent: Tuesday, October 23, 2012 5:46:57 PM >> Subject: [Swift-user] running swift k on multiple instances on EC2, using coaster-service >> Hi everyone >> >> >> I am trying to run a Montage application workflow with swift on >> multiple instances of AMAZON EC2. >> So far I was able to set up a cluster, and a PVFS files system shared >> among the nodes ( using FUSE. so I will have POSIX interface on my >> swift work directory ). >> I have tried running a simple hello.swift example on multiple nodes >> with the coaster. the working directory is the shared folder >> (supported by PVFS). >> when I run the code using my own tc.data and sites.xml, this will >> happen: >> >> >> >> (my command) ubuntu at ip-10-244-4-101:~/coaster$ swift -tc.file tc.data >> -sites.file sites.xml ~/swift-0.93/examples/swift/tutorial/hello.swift >> (results:) >> Swift 0.93 swift-r5483 cog-r3339 >> >> >> RunID: 20121023-2200-4d3knr72 >> Progress: time: Tue, 23 Oct 2012 22:00:50 +0000 >> Find: http://10.244.4.101:1213 >> Find: keepalive(120), reconnect - http://10.244.4.101:1213 >> Passive queue processor initialized. Callback URI is >> http://10.244.4.101:1212 >> Progress: time: Tue, 23 Oct 2012 22:01:20 +0000 Submitted:1 >> Progress: time: Tue, 23 Oct 2012 22:01:50 +0000 Submitted:1 >> Progress: time: Tue, 23 Oct 2012 22:02:20 +0000 Submitted:1 >> Progress: time: Tue, 23 Oct 2012 22:02:50 +0000 Submitted:1 >> Progress: time: Tue, 23 Oct 2012 22:03:20 +0000 Submitted:1 >> Progress: time: Tue, 23 Oct 2012 22:03:50 +0000 Submitted:1 >> Progress: time: Tue, 23 Oct 2012 22:04:20 +0000 Submitted:1 >> >> >> and it keeps doing this forever meaning that there is no answer from >> worker nodes! >> as I checked on worker nodes, the working files are created on the >> shared folder, and when i check the running applications, there is a >> java application running. but nothing happens. >> I have also attached the log file of my hello.swift running in case >> you need to take a look at it. >> should I consider using pbs, or condor,... I have no idea about how >> they work though. >> >> >> I appreciate if anyone can help me with it. Thank you so much. >> >> Best, -- >> Iman Sadooghi >> Illinois Institute of Technology (IIT) >> Data-Intensive Distributed Systems Laboratory >> >> >> _______________________________________________ >> Swift-user mailing list >> Swift-user at ci.uchicago.edu >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ ================================================================= ================================================================= From ketancmaheshwari at gmail.com Wed Oct 24 12:11:04 2012 From: ketancmaheshwari at gmail.com (Ketan Maheshwari) Date: Wed, 24 Oct 2012 13:11:04 -0400 Subject: [Swift-user] running swift k on multiple instances on EC2, using coaster-service In-Reply-To: <50881B95.6070202@cs.iit.edu> References: <1925150560.162848.1351092530047.JavaMail.root@zimbra-mb2.anl.gov> <50881B95.6070202@cs.iit.edu> Message-ID: Hi Ioan, I've been extensively using Swift on Amazon EC2 via coasters. I am using a fedora 8 image. I use an external host as a submit host so, only perl (for worker.pl; is installed) and application binaries needs to be on EC2 nodes. The name of my image is ketan_mars and is on us-east-1 region. Let me know if you have more questions in this regard. Regards, Ketan On Wed, Oct 24, 2012 at 12:47 PM, Ioan Raicu wrote: > Hi David, > Has anyone in the Swift group run Swift on Amazon? Perhaps there are > already some pre-configured images Iman can use, that already has the > environment (Linux, Java, Swift, PBS/Condor, GridFTP, NFS/PVFS, etc) > setup and ready to use. If there are such images, please point Iman in > the right direction. If not, perhaps after Iman finishes his > configuration, we can reference his images for others to use on Amazon. > Ioan > > On 10/24/2012 10:28 AM, David Kelly wrote: > > Iman, > > > > There is some documentation on how to use the start-coaster-service > script at > http://www.ci.uchicago.edu/swift/guides/release-0.93/siteguide/siteguide.html#_bag_of_workstations. > You should be able to follow the "bag of workstations" configuration > example once you have started your EC2 instance. That may help to simplify > things for you a bit. Please let me know if you have any questions or > issues with this. > > > > Thanks, > > David > > > > ----- Original Message ----- > >> From: "Iman Sadooghi" > >> To: swift-user at ci.uchicago.edu > >> Sent: Tuesday, October 23, 2012 5:46:57 PM > >> Subject: [Swift-user] running swift k on multiple instances on EC2, > using coaster-service > >> Hi everyone > >> > >> > >> I am trying to run a Montage application workflow with swift on > >> multiple instances of AMAZON EC2. > >> So far I was able to set up a cluster, and a PVFS files system shared > >> among the nodes ( using FUSE. so I will have POSIX interface on my > >> swift work directory ). > >> I have tried running a simple hello.swift example on multiple nodes > >> with the coaster. the working directory is the shared folder > >> (supported by PVFS). > >> when I run the code using my own tc.data and sites.xml, this will > >> happen: > >> > >> > >> > >> (my command) ubuntu at ip-10-244-4-101:~/coaster$ swift -tc.file tc.data > >> -sites.file sites.xml ~/swift-0.93/examples/swift/tutorial/hello.swift > >> (results:) > >> Swift 0.93 swift-r5483 cog-r3339 > >> > >> > >> RunID: 20121023-2200-4d3knr72 > >> Progress: time: Tue, 23 Oct 2012 22:00:50 +0000 > >> Find: http://10.244.4.101:1213 > >> Find: keepalive(120), reconnect - http://10.244.4.101:1213 > >> Passive queue processor initialized. Callback URI is > >> http://10.244.4.101:1212 > >> Progress: time: Tue, 23 Oct 2012 22:01:20 +0000 Submitted:1 > >> Progress: time: Tue, 23 Oct 2012 22:01:50 +0000 Submitted:1 > >> Progress: time: Tue, 23 Oct 2012 22:02:20 +0000 Submitted:1 > >> Progress: time: Tue, 23 Oct 2012 22:02:50 +0000 Submitted:1 > >> Progress: time: Tue, 23 Oct 2012 22:03:20 +0000 Submitted:1 > >> Progress: time: Tue, 23 Oct 2012 22:03:50 +0000 Submitted:1 > >> Progress: time: Tue, 23 Oct 2012 22:04:20 +0000 Submitted:1 > >> > >> > >> and it keeps doing this forever meaning that there is no answer from > >> worker nodes! > >> as I checked on worker nodes, the working files are created on the > >> shared folder, and when i check the running applications, there is a > >> java application running. but nothing happens. > >> I have also attached the log file of my hello.swift running in case > >> you need to take a look at it. > >> should I consider using pbs, or condor,... I have no idea about how > >> they work though. > >> > >> > >> I appreciate if anyone can help me with it. Thank you so much. > >> > >> Best, -- > >> Iman Sadooghi > >> Illinois Institute of Technology (IIT) > >> Data-Intensive Distributed Systems Laboratory > >> > >> > >> _______________________________________________ > >> Swift-user mailing list > >> Swift-user at ci.uchicago.edu > >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > _______________________________________________ > > Swift-user mailing list > > Swift-user at ci.uchicago.edu > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > -- > ================================================================= > Ioan Raicu, Ph.D. > Assistant Professor, Illinois Institute of Technology (IIT) > Guest Research Faculty, Argonne National Laboratory (ANL) > ================================================================= > Data-Intensive Distributed Systems Laboratory, CS/IIT > Distributed Systems Laboratory, MCS/ANL > ================================================================= > Cel: 1-847-722-0876 > Office: 1-312-567-5704 > Email: iraicu at cs.iit.edu > Web: http://www.cs.iit.edu/~iraicu/ > Web: http://datasys.cs.iit.edu/ > ================================================================= > ================================================================= > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > -- Ketan -------------- next part -------------- An HTML attachment was scrubbed... URL: From iraicu at cs.iit.edu Wed Oct 24 12:13:06 2012 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Wed, 24 Oct 2012 12:13:06 -0500 Subject: [Swift-user] running swift k on multiple instances on EC2, using coaster-service In-Reply-To: References: <1925150560.162848.1351092530047.JavaMail.root@zimbra-mb2.anl.gov> <50881B95.6070202@cs.iit.edu> Message-ID: <5B84EE56-8D3B-4D22-880E-73EC9DBF2F77@cs.iit.edu> That sounds great! Iman, can you give Ketan's image a try, and report back if it worked, or any other problems encountered? Ioan -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor ================================================================= Computer Science Department Illinois Institute of Technology 10 W. 31st Street Chicago, IL 60616 ================================================================= Cel: 1-847-722-0876 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ ================================================================= ================================================================= On Oct 24, 2012, at 12:11 PM, Ketan Maheshwari wrote: > Hi Ioan, > > I've been extensively using Swift on Amazon EC2 via coasters. > > I am using a fedora 8 image. > > I use an external host as a submit host so, only perl (for worker.pl; is installed) and application binaries needs to be on EC2 nodes. > > The name of my image is ketan_mars and is on us-east-1 region. > > Let me know if you have more questions in this regard. > > Regards, > Ketan > > On Wed, Oct 24, 2012 at 12:47 PM, Ioan Raicu wrote: >> Hi David, >> Has anyone in the Swift group run Swift on Amazon? Perhaps there are >> already some pre-configured images Iman can use, that already has the >> environment (Linux, Java, Swift, PBS/Condor, GridFTP, NFS/PVFS, etc) >> setup and ready to use. If there are such images, please point Iman in >> the right direction. If not, perhaps after Iman finishes his >> configuration, we can reference his images for others to use on Amazon. >> Ioan >> >> On 10/24/2012 10:28 AM, David Kelly wrote: >> > Iman, >> > >> > There is some documentation on how to use the start-coaster-service script at http://www.ci.uchicago.edu/swift/guides/release-0.93/siteguide/siteguide.html#_bag_of_workstations. You should be able to follow the "bag of workstations" configuration example once you have started your EC2 instance. That may help to simplify things for you a bit. Please let me know if you have any questions or issues with this. >> > >> > Thanks, >> > David >> > >> > ----- Original Message ----- >> >> From: "Iman Sadooghi" >> >> To: swift-user at ci.uchicago.edu >> >> Sent: Tuesday, October 23, 2012 5:46:57 PM >> >> Subject: [Swift-user] running swift k on multiple instances on EC2, using coaster-service >> >> Hi everyone >> >> >> >> >> >> I am trying to run a Montage application workflow with swift on >> >> multiple instances of AMAZON EC2. >> >> So far I was able to set up a cluster, and a PVFS files system shared >> >> among the nodes ( using FUSE. so I will have POSIX interface on my >> >> swift work directory ). >> >> I have tried running a simple hello.swift example on multiple nodes >> >> with the coaster. the working directory is the shared folder >> >> (supported by PVFS). >> >> when I run the code using my own tc.data and sites.xml, this will >> >> happen: >> >> >> >> >> >> >> >> (my command) ubuntu at ip-10-244-4-101:~/coaster$ swift -tc.file tc.data >> >> -sites.file sites.xml ~/swift-0.93/examples/swift/tutorial/hello.swift >> >> (results:) >> >> Swift 0.93 swift-r5483 cog-r3339 >> >> >> >> >> >> RunID: 20121023-2200-4d3knr72 >> >> Progress: time: Tue, 23 Oct 2012 22:00:50 +0000 >> >> Find: http://10.244.4.101:1213 >> >> Find: keepalive(120), reconnect - http://10.244.4.101:1213 >> >> Passive queue processor initialized. Callback URI is >> >> http://10.244.4.101:1212 >> >> Progress: time: Tue, 23 Oct 2012 22:01:20 +0000 Submitted:1 >> >> Progress: time: Tue, 23 Oct 2012 22:01:50 +0000 Submitted:1 >> >> Progress: time: Tue, 23 Oct 2012 22:02:20 +0000 Submitted:1 >> >> Progress: time: Tue, 23 Oct 2012 22:02:50 +0000 Submitted:1 >> >> Progress: time: Tue, 23 Oct 2012 22:03:20 +0000 Submitted:1 >> >> Progress: time: Tue, 23 Oct 2012 22:03:50 +0000 Submitted:1 >> >> Progress: time: Tue, 23 Oct 2012 22:04:20 +0000 Submitted:1 >> >> >> >> >> >> and it keeps doing this forever meaning that there is no answer from >> >> worker nodes! >> >> as I checked on worker nodes, the working files are created on the >> >> shared folder, and when i check the running applications, there is a >> >> java application running. but nothing happens. >> >> I have also attached the log file of my hello.swift running in case >> >> you need to take a look at it. >> >> should I consider using pbs, or condor,... I have no idea about how >> >> they work though. >> >> >> >> >> >> I appreciate if anyone can help me with it. Thank you so much. >> >> >> >> Best, -- >> >> Iman Sadooghi >> >> Illinois Institute of Technology (IIT) >> >> Data-Intensive Distributed Systems Laboratory >> >> >> >> >> >> _______________________________________________ >> >> Swift-user mailing list >> >> Swift-user at ci.uchicago.edu >> >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user >> > _______________________________________________ >> > Swift-user mailing list >> > Swift-user at ci.uchicago.edu >> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user >> >> -- >> ================================================================= >> Ioan Raicu, Ph.D. >> Assistant Professor, Illinois Institute of Technology (IIT) >> Guest Research Faculty, Argonne National Laboratory (ANL) >> ================================================================= >> Data-Intensive Distributed Systems Laboratory, CS/IIT >> Distributed Systems Laboratory, MCS/ANL >> ================================================================= >> Cel: 1-847-722-0876 >> Office: 1-312-567-5704 >> Email: iraicu at cs.iit.edu >> Web: http://www.cs.iit.edu/~iraicu/ >> Web: http://datasys.cs.iit.edu/ >> ================================================================= >> ================================================================= >> >> _______________________________________________ >> Swift-user mailing list >> Swift-user at ci.uchicago.edu >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > > > -- > Ketan > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ketancmaheshwari at gmail.com Wed Oct 24 12:22:18 2012 From: ketancmaheshwari at gmail.com (Ketan Maheshwari) Date: Wed, 24 Oct 2012 13:22:18 -0400 Subject: [Swift-user] running swift k on multiple instances on EC2, using coaster-service In-Reply-To: <5B84EE56-8D3B-4D22-880E-73EC9DBF2F77@cs.iit.edu> References: <1925150560.162848.1351092530047.JavaMail.root@zimbra-mb2.anl.gov> <50881B95.6070202@cs.iit.edu> <5B84EE56-8D3B-4D22-880E-73EC9DBF2F77@cs.iit.edu> Message-ID: It is ketan-mars and ami id is: ami-1a1ca273 On Wed, Oct 24, 2012 at 1:13 PM, Ioan Raicu wrote: > That sounds great! Iman, can you give Ketan's image a try, and report back > if it worked, or any other problems encountered? > > Ioan > > -- > ================================================================= > Ioan Raicu, Ph.D. > Assistant Professor > ================================================================= > Computer Science Department > Illinois Institute of Technology > 10 W. 31st Street Chicago, IL 60616 > ================================================================= > Cel: 1-847-722-0876 > Email: iraicu at cs.iit.edu > Web: http://www.cs.iit.edu/~iraicu/ > ================================================================= > ================================================================= > > > > On Oct 24, 2012, at 12:11 PM, Ketan Maheshwari > wrote: > > Hi Ioan, > > I've been extensively using Swift on Amazon EC2 via coasters. > > I am using a fedora 8 image. > > I use an external host as a submit host so, only perl (for worker.pl; is > installed) and application binaries needs to be on EC2 nodes. > > The name of my image is ketan_mars and is on us-east-1 region. > > Let me know if you have more questions in this regard. > > Regards, > Ketan > > On Wed, Oct 24, 2012 at 12:47 PM, Ioan Raicu wrote: > >> Hi David, >> Has anyone in the Swift group run Swift on Amazon? Perhaps there are >> already some pre-configured images Iman can use, that already has the >> environment (Linux, Java, Swift, PBS/Condor, GridFTP, NFS/PVFS, etc) >> setup and ready to use. If there are such images, please point Iman in >> the right direction. If not, perhaps after Iman finishes his >> configuration, we can reference his images for others to use on Amazon. >> Ioan >> >> On 10/24/2012 10:28 AM, David Kelly wrote: >> > Iman, >> > >> > There is some documentation on how to use the start-coaster-service >> script at >> http://www.ci.uchicago.edu/swift/guides/release-0.93/siteguide/siteguide.html#_bag_of_workstations. >> You should be able to follow the "bag of workstations" configuration >> example once you have started your EC2 instance. That may help to simplify >> things for you a bit. Please let me know if you have any questions or >> issues with this. >> > >> > Thanks, >> > David >> > >> > ----- Original Message ----- >> >> From: "Iman Sadooghi" >> >> To: swift-user at ci.uchicago.edu >> >> Sent: Tuesday, October 23, 2012 5:46:57 PM >> >> Subject: [Swift-user] running swift k on multiple instances on EC2, >> using coaster-service >> >> Hi everyone >> >> >> >> >> >> I am trying to run a Montage application workflow with swift on >> >> multiple instances of AMAZON EC2. >> >> So far I was able to set up a cluster, and a PVFS files system shared >> >> among the nodes ( using FUSE. so I will have POSIX interface on my >> >> swift work directory ). >> >> I have tried running a simple hello.swift example on multiple nodes >> >> with the coaster. the working directory is the shared folder >> >> (supported by PVFS). >> >> when I run the code using my own tc.data and sites.xml, this will >> >> happen: >> >> >> >> >> >> >> >> (my command) ubuntu at ip-10-244-4-101:~/coaster$ swift -tc.file tc.data >> >> -sites.file sites.xml ~/swift-0.93/examples/swift/tutorial/hello.swift >> >> (results:) >> >> Swift 0.93 swift-r5483 cog-r3339 >> >> >> >> >> >> RunID: 20121023-2200-4d3knr72 >> >> Progress: time: Tue, 23 Oct 2012 22:00:50 +0000 >> >> Find: http://10.244.4.101:1213 >> >> Find: keepalive(120), reconnect - http://10.244.4.101:1213 >> >> Passive queue processor initialized. Callback URI is >> >> http://10.244.4.101:1212 >> >> Progress: time: Tue, 23 Oct 2012 22:01:20 +0000 Submitted:1 >> >> Progress: time: Tue, 23 Oct 2012 22:01:50 +0000 Submitted:1 >> >> Progress: time: Tue, 23 Oct 2012 22:02:20 +0000 Submitted:1 >> >> Progress: time: Tue, 23 Oct 2012 22:02:50 +0000 Submitted:1 >> >> Progress: time: Tue, 23 Oct 2012 22:03:20 +0000 Submitted:1 >> >> Progress: time: Tue, 23 Oct 2012 22:03:50 +0000 Submitted:1 >> >> Progress: time: Tue, 23 Oct 2012 22:04:20 +0000 Submitted:1 >> >> >> >> >> >> and it keeps doing this forever meaning that there is no answer from >> >> worker nodes! >> >> as I checked on worker nodes, the working files are created on the >> >> shared folder, and when i check the running applications, there is a >> >> java application running. but nothing happens. >> >> I have also attached the log file of my hello.swift running in case >> >> you need to take a look at it. >> >> should I consider using pbs, or condor,... I have no idea about how >> >> they work though. >> >> >> >> >> >> I appreciate if anyone can help me with it. Thank you so much. >> >> >> >> Best, -- >> >> Iman Sadooghi >> >> Illinois Institute of Technology (IIT) >> >> Data-Intensive Distributed Systems Laboratory >> >> >> >> >> >> _______________________________________________ >> >> Swift-user mailing list >> >> Swift-user at ci.uchicago.edu >> >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user >> > _______________________________________________ >> > Swift-user mailing list >> > Swift-user at ci.uchicago.edu >> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user >> >> -- >> ================================================================= >> Ioan Raicu, Ph.D. >> Assistant Professor, Illinois Institute of Technology (IIT) >> Guest Research Faculty, Argonne National Laboratory (ANL) >> ================================================================= >> Data-Intensive Distributed Systems Laboratory, CS/IIT >> Distributed Systems Laboratory, MCS/ANL >> ================================================================= >> Cel: 1-847-722-0876 >> Office: 1-312-567-5704 >> Email: iraicu at cs.iit.edu >> Web: http://www.cs.iit.edu/~iraicu/ >> Web: http://datasys.cs.iit.edu/ >> ================================================================= >> ================================================================= >> >> _______________________________________________ >> Swift-user mailing list >> Swift-user at ci.uchicago.edu >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user >> > > > > -- > Ketan > > > -- Ketan -------------- next part -------------- An HTML attachment was scrubbed... URL: From iraicu at cs.iit.edu Fri Oct 26 12:43:40 2012 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Fri, 26 Oct 2012 13:43:40 -0400 Subject: [Swift-user] Call for Workshops: ACM HPDC 2013 -- deadline extended to November 1, 2012 Message-ID: <508ACBCC.706@cs.iit.edu> Call for Workshops The organizers of the /22nd International ACM Symposium on High-Performance Parallel and Distributed Computing/ (HPDC'13) *call for proposals for workshops* to be held with HPDC'13. The workshops will be held on June 17-18, 2013. Workshops should provide forums for discussion among researchers and practitioners on focused topics or emerging research areas relevant to the HPDC community. Organizers may structure workshops as they see fit, including invited talks, panel discussions, presentations of work in progress, fully peer-reviewed papers, or some combination. Workshops could be scheduled for half a day or a full day, depending on interest, space constraints, and organizer preference. Organizers should design workshops for approximately 20-40 participants, to balance impact and effective discussion. *Workshop proposals* must be sent in PDF format to the HPDC'13 Workshops Chair, Abhishek Chandra (Email: chandra AT cs DOT umn DOT edu ) with the subject line *"HPDC 2013 Workshop Proposal"*, and should include: * The name and acronym of the workshop * A description (0.5-1 page) of the theme of the workshop * A description (one paragraph) of the relation between the theme of the workshop and of HPDC * A list of topics of interest * The names and affiliations of the workshop organizers, and if applicable, of a significant portion of the program committee * A description of the expected structure of the workshop (papers, invited talks, panel discussions, etc.) * Data about previous offerings of the workshop (if any), including the attendance, the numbers of papers or presentations submitted and accepted, and the links to the corresponding websites * A publicity plan for attracting submissions and attendees. Please also include expected number of submissions, accepted papers, and attendees that you anticipate for a successful workshop. Due to publication deadlines, workshops must operate within roughly the following timeline: papers due mid February (2-3 weeks after the HPDC deadline), and selected and sent to the publisher by mid April. Important dates: *Workshop Proposals Due: * *November 1, 2012* Notifications: November 7, 2012 Workshop CFPs Online and Distributed: November 25, 2012 -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From iraicu at cs.iit.edu Fri Oct 26 13:13:51 2012 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Fri, 26 Oct 2012 14:13:51 -0400 Subject: [Swift-user] CFP: 22nd Int. ACM Symposium on High-Performance Parallel and Distributed Computing (HPDC'13) Message-ID: <508AD2DF.8060609@cs.iit.edu> **** CALL FOR PAPERS **** The 22nd International ACM Symposium on High-Performance Parallel and Distributed Computing (HPDC'13) New York City, USA - June 17-21, 2013 http://www.hpdc.org/2013 The ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) is the premier annual conference for presenting the latest research on the design, implementation, evaluation, and the use of parallel and distributed systems for high-end computing. In 2013, the 22nd HPDC and affiliated workshops will take place in the heart of iconic New York City from June 17-21. **** SUBMISSION DEADLINES **** Abstracts: 14 January 2013 Papers: 21 January 2013 (no extensions) **** HPDC'13 GENERAL CO-CHAIRS **** Manish Parashar, Rutgers University Jon Weissman, University of Minnesota **** HPDC'13 PROGRAM CO-CHAIRS **** Dick Epema, Delft University of Technology Renato Figueiredo, University of Florida **** HPDC'13 WORKSHOPS CHAIR **** Abhishek Chandra, University of Minnesota **** HPDC'13 LOCAL ARRANGEMENTS CHAIR **** Daniele Scarpazza, DEShaw Research **** HPDC'13 SPONSORSHIP CHAIR **** Dean Hildebrand, IBM Almaden **** HPDC'13 PUBLICITY CO-CHAIRS **** Alexandru Iosup, Delft University of Technology, the Netherlands Ioan Raicu, Illinois Institute of Technology, USA Kenjiro Taura, University of Tokyo, Japan Bruno Schulze, National Laboratory for Scientific Computing, Brazil **** SCOPE AND TOPICS **** Submissions are welcomed on high-performance parallel and distributed computing topics including but not limited to: clusters, clouds, grids, data-intensive computing, massively multicore, and global-scale computing systems. New scholarly research showing empirical and reproducible results in architectures, systems, and networks is strongly encouraged, as are experience reports of operational deployments that can provide insights for future research on HPDC applications and systems. All papers will be evaluated for their originality, technical depth and correctness, potential impact, relevance to the conference, and quality of presentation. Research papers must clearly demonstrate research contributions and novelty, while experience reports must clearly describe lessons learned and demonstrate impact. In the context of high-?????performance parallel and distributed computing, the topics of interest include, but are not limited to: * Systems, networks, and architectures for high-end computing * Massively multicore systems * Resource virtualization * Programming languages and environments * I/O, storage systems, and data management * Resource management and scheduling, including energy-?????aware techniques * Performance modeling and analysis * Fault tolerance, reliability, and availability * Data-intensive computing * Applications of parallel and distributed computing **** PAPER SUBMISSION GUIDELINES **** Authors are invited to submit technical papers of at most 12 pages in PDF format, including figures and references. Papers should be formatted in the ACM Proceedings Style and submitted via the conference web site. No changes to the margins, spacing, or font sizes as specified by the style file are allowed. Accepted papers will appear in the conference proceedings, and will be incorporated into the ACM Digital Library. A limited number of papers will be accepted as posters. Papers must be self-contained and provide the technical substance required for the program committee to evaluate their contributions. Papers should thoughtfully address all related work, particularly work presented at previous HPDC events. Submitted papers must be original work that has not appeared in and is not under consideration for another conference or a journal. See the ACM Prior Publication Policy for more details. **** IMPORTANT DATES **** Abstracts Due: 14 January 2013 Papers Due: 21 January 2013 (no extensions) **** Program Committee **** David Abramson, Monash University, Australia Kento Aida, National Institute of Informatics, Japan Gabriel Antoniu INRIA, France Henri Bal, Vrije Universiteit, the Netherlands Adam Barker, University of St Andrews, UK Michela Becchi, University of Missouri - Columbia, USA John Bent, EMC, USA Ali Butt, Virginia Tech, USA Kirk Cameron, Virginia Tech, USA Franck Cappello, INRIA, France and University of Illinois at Urbana-Champaign, USA Henri Casanova, University of Hawaii, USA Abhishek Chandra, University of Minnesota, USA Andrew Chien, University of Chicago and Argonne National Laboratory, USA Paolo Costa, Imperial College London, UK Peter Dinda, Northwestern University, USA Gilles Fedak, INRIA, France Ian Foster, University of Chicago and Argonne National Laboratory, USA Clemens Grelck, University of Amsterdam, the Netherlands Dean Hildebrand, IBM Research, USA Fabrice Huet, INRIA-University of Nice, France Adriana Iamnitchi, University of South Florida, USA Alexandru Iosup, Delft University of Technology, the Netherlands Kate Keahey, Argonne National Laboratory, USA Thilo Kielmann, Vrije Universiteit, the Netherlands Charles Kilian, Purdue University, USA Zhiling Lan, Illinois Institute of Technology, USA John Lange, University of Pittsburgh, USA Barney Maccabe, Oak Ridge National Laboratory, USA Carlos Maltzahn, University of California, Santa Cruz, USA Naoya Maruyama, RIKEN Advanced Institute for Computational Science, Japan Satoshi Matsuoka, Tokyo Institute of Technology, Japan Manish Parashar, Rutgers University, USA Judy Qiu, Indiana University, USA Ioan Raicu, Illinois Institute of Technology, USA Philip Rhodes, University of Mississippi, USA Matei Ripeanu, University of British Columbia, Canada Prasenjit Sarkar, IBM Research, USA Daniele Scarpazza, D.E. Shaw Research, USA Karsten Schwan, Georgia Institute of Technology, USA Martin Swany, Indiana University, USA Michela Taufer, University of Delaware, USA Kenjiro Taura, University of Tokyo, Japan Douglas Thain, University of Notre Dame, USA Cristian Ungureanu, NEC Research, USA Ana Varbanescu, Delft University of Technology, the Netherlands Chuliang Weng, Shanghai Jiao Tong University, China Jon Weissman, University of Minnesota, USA Yongwei Wu, Tsinghua University, China Dongyan Xu, Purdue University, USA Ming Zhao, Florida International University, USA **** Steering Committee **** Henri Bal, Vrije Universiteit, the Netherlands Andrew A. Chien, University of Chicago and Argonne National Laboratory, USA Peter Dinda, Northwestern University, USA Dick Epema, Delft University of Technology, the Netherlands Ian Foster, University of Chicago and Argonne National Laboratory, USA Salim Hariri, University of Arizona, USA Thilo Kielmann, Vrije Universiteit, the Netherlands Dieter Kranzlmueller, Ludwig-Maximilians-Universitaet Muenchen, Germany Arthur "Barney" Maccabe, Oak Ridge National Laboratory, USA Satoshi Matsuoka, Tokyo Institute of Technology, Japan Manish Parashar, Rutgers University, USA Matei Ripeanu, University of British Columbia, Canada Karsten Schwan, Georgia Tech, USA Doug Thain, University of Notre Dame, USA Jon Weissman, University of Minnesota (Chair), USA -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ ================================================================= ================================================================= From iraicu at cs.iit.edu Fri Oct 26 13:26:57 2012 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Fri, 26 Oct 2012 14:26:57 -0400 Subject: [Swift-user] CFP: The 13th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid 2013) Message-ID: <508AD5F1.2020301@cs.iit.edu> **** CALL FOR PAPERS **** The 13th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid 2013) Delft University of Technology, Delft, the Netherlands May 13-16, 2013 http://www.pds.ewi.tudelft.nl/ccgrid2013 Rapid advances in architectures, networks, and systems and middleware technologies are leading to new concepts in and platforms for computing, ranging from Clusters and Grids to Clouds and Datacenters. CCGrid is a series of very successful conferences, sponsored by the IEEE Computer Society Technical Committee on Scalable Computing (TCSC) and the ACM, with the overarching goal of bringing together international researchers, developers, and users to provide an international forum to present leading research activities and results on a broad range of topics related to these concepts and platforms, and their applications. The conference features keynotes, technical presentations, workshops, tutorials, and posters, as well as the SCALE challenge featuring live demonstrations. In 2013, CCGrid will come to the Netherlands for the first time, and will be held in Delft, a historical, picturesque city that is less than one hour away from Amsterdam-Schiphol airport. The main conference will be held on May 14-16 (Tuesday to Thursday), with tutorials and affiliated workshops taking place on May 13 (Monday). **** IMPORTANT DATES **** Papers Due: 12 November 2012 Author Notifications: 24 January 2013 Final Papers Due: 22 February 2013 **** TOPICS OF INTEREST **** CCGrid 2013 will have a focus on important and immediate issues that are significantly influencing all aspects of cluster, cloud and grid computing. Topics of interest include, but are not limited to: * Applications and Experiences: Applications to real and complex problems in science, engineering, business, and society; User studies; Experiences with large-scale deployments, systems, or applications * Architecture: System architectures, design and deployment; Power and cooling; Security and reliability; High availability solutions * Autonomic Computing and Cyberinfrastructure: Self-managed behavior, models and technologies; Autonomic paradigms and systems (control-based, bio-inspired, emergent, etc.); Bio-inspired optimizations and computing * Cloud Computing: Cloud architectures; Software tools and techniques for clouds * Multicore and Accelerator-based Computing: Software and application techniques to utilize multicore architectures and accelerators in clusters, grids, and clouds * Performance Modeling and Evaluation: Performance prediction and modeling; Monitoring and evaluation tools; Analysis of system and application performance; Benchmarks and testbeds * Programming Models, Systems, and Fault-Tolerant Computing: Programming models and environments for cluster, cloud, and grid computing; Fault-tolerant systems, programs and algorithms; Systems software to support efficient computing * Scheduling and Resource Management: Techniques to schedule jobs and resources on cluster, cloud, and grid computing platforms; SLA definition and enforcement **** PAPER SUBMISSION GUIDELINES **** Authors are invited to submit papers electronically in PDF format. Submitted manuscripts should be structured as technical papers and may not exceed 8 letter-size (8.5 x 11) pages including figures, tables and references using the IEEE format for conference proceedings. Submissions not conforming to these guidelines may be returned without review. Authors should make sure that their file will print on a printer that uses letter-size (8.5 x 11) paper. The official language of the conference is English. All manuscripts will be reviewed and will be judged on correctness, originality, technical strength, significance, quality of presentation, and interest and relevance to the conference attendees. Submitted papers must represent original unpublished research that is not currently under review for any other conference or journal. Papers not following these guidelines will be rejected without review and further action may be taken, including (but not limited to) notifications sent to the heads of the institutions of the authors and sponsors of the conference. Submissions received after the due date, exceeding the page limit, or not appropriately structured may not be considered. Authors may contact the conference chairs for more information. The proceedings will be published through the IEEE Computer Society Press, USA, and will be made available online through the IEEE Digital Library. **** CALL FOR TUTORIAL AND WORKSHOP PROPOSALS **** Tutorials and workshops affiliated with CCGrid 2013 will be held on May 13 (Monday). For more information on the tutorials and workshops and for the complete Call for Tutorial and Workshop Proposals, please see the conference website. **** GENERAL CHAIR **** Dick Epema, Delft University of Technology, the Netherlands **** PROGRAM CHAIR **** Thomas Fahringer, University of Innsbruck, Austria **** PROGRAM VICE-CHAIRS **** Rosa Badia, Barcelona Supercomputing Center, Spain Henri Bal, Vrije Universiteit, the Netherlands Marios Dikaiakos, University of Cyprus, Cyprus Kirk Cameron, VirginiaTech, USA Daniel Katz, University of Chicago & Argonne Nat Lab, USA Kate Keahey, Argonne National Laboratory, USA Martin Schulz, Lawrence Livermore National Laboratory, USA Douglas Thain, University of Notre Dame, USA Cheng-Zhong Xu, Shenzhen Inst. of Advanced Techn, China **** WORKSHOPS CO-CHAIRS **** Shantenu Jha, Rutgers and Louisana State University, USA Ioan Raicu, Illinois Institute of Technology, USA **** TOTORIALS CHAIR **** Radu Prodan, University of Innsbruck, Austria **** DOCTORAL SYMPOSIUM CO-CHAIRS **** Yogesh Simmhan, University of Southern California, USA Ana Varbanescu, Delft Univ of Technology, the Netherlands **** SUBMISSIONS AND PROCEEDINGS CHAIR **** Pavan Balaji, Argonne National Laboratory, USA **** FINANCE AND REGISTRATION CHAIR **** Alexandru Iosup, Delft Univ of Technology, the Netherlands **** PUBLICITY CHAIRS **** Nazareno Andrade, University Federal de Campina Grance, Brazil Gabriel Antoniu, INRIA, France Bahman Javadi, University of Western Sysney, Australia Ioan Raicu, Illinois Institute of Technology and Argonne National Laboratory, USA Kin Choong Yow, Shenzhen Inst. of Advanced Technology, China **** CYBER CHAIR **** Stephen van der Laan, Delft University of Technology, the Netherlands -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ ================================================================= ================================================================= From yanghao0614 at gmail.com Tue Oct 30 17:50:20 2012 From: yanghao0614 at gmail.com (Hao Yang) Date: Tue, 30 Oct 2012 15:50:20 -0700 Subject: [Swift-user] output data type of app procedures Message-ID: Hi, In Swift/K, is it possible to have int type or string type as the output data of an app procedure? Thank you. -- Best Regards, Hao Yang Networked Systems Laboratory Electrical and Computer Engineering The University of British Columbia -------------- next part -------------- An HTML attachment was scrubbed... URL: From wilde at mcs.anl.gov Tue Oct 30 17:53:04 2012 From: wilde at mcs.anl.gov (Michael Wilde) Date: Tue, 30 Oct 2012 17:53:04 -0500 (CDT) Subject: [Swift-user] output data type of app procedures In-Reply-To: Message-ID: <1774078846.7396.1351637584976.JavaMail.root@zimbra.anl.gov> Hao, The way to do this is have the app() (typically a script) write one or more such values into a file, and then use one of the forms of the readData() function to extract the data from the file into Swift variables. - Mike ----- Original Message ----- > From: "Hao Yang" > To: swift-user at ci.uchicago.edu > Sent: Tuesday, October 30, 2012 5:50:20 PM > Subject: [Swift-user] output data type of app procedures > Hi, > > > In Swift/K, is it possible to have int type or string type as the > output data of an app procedure? > > > Thank you. > > -- > Best Regards, > Hao Yang > Networked Systems Laboratory > Electrical and Computer Engineering > The University of British Columbia > > > > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory From isadoogh at iit.edu Mon Oct 22 11:28:38 2012 From: isadoogh at iit.edu (Iman Sadooghi) Date: Mon, 22 Oct 2012 16:28:38 -0000 Subject: [Swift-user] working with swift on Amazon EC2 Message-ID: Hi Everyone I'm trying to make swift working on multiple instances of Amazon EC2. so I can run Montage with Swift. For that I was trying to set up PVFS on the instances. So far I could not do it. my problem is setting up the FUSE enabled PVFS so I can run my project on the PVFS mount point. I'm using Coaster-service to run all these. I have also tried with the option of (SHARED_FILESYSTEM=NO) not using shared file system. that didn't work either. I think the problem is setting up a PVFS mount point that supports POSIX operations. Is this possible at all? any comment would help. Thank you in advnace -- Iman Sadooghi Illinois Institute of Technology (IIT) Data-Intensive Distributed Systems Laboratory -------------- next part -------------- An HTML attachment was scrubbed... URL: