From tianyu491433909 at 163.com Wed Jun 1 01:40:06 2011 From: tianyu491433909 at 163.com (tianyu491433909) Date: Wed, 1 Jun 2011 14:40:06 +0800 (CST) Subject: [Swift-user] a cscript.exe error Message-ID: <4b138338.7dbb.13049ee38b6.Coremail.tianyu491433909@163.com> I have problems using the windows environment? cscript.exe //Nologo shared\_swiftwrap.vbs wc-52333yak -jobdir 5 -scratch -e C:/swift/usr/local/wbin/wc.exe -out count14.txt -err stderr.txt -i -d _concurrent| -if _concurrent/outfile-05660c60-e9ce-4b50-8e09-31392aed6957- -of count14.txt -k -status files -a -w _concurrent/outfile-05660c60-e9ce-4b50-8e09-31392aed6957- '-if' is not an internal or external command, operable program or batch file. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tianyu491433909 at 163.com Thu Jun 2 03:33:45 2011 From: tianyu491433909 at 163.com (tianyu491433909) Date: Thu, 2 Jun 2011 16:33:45 +0800 (CST) Subject: [Swift-user] a cscript.exe error Message-ID: <42b199.1194d.1304f7ca0cc.Coremail.tianyu491433909@163.com> I have problems using the windows environment? cscript.exe //Nologo shared\_swiftwrap.vbs wc-52333yak -jobdir 5 -scratch -e C:/swift/usr/local/wbin/wc.exe -out count14.txt -err stderr.txt -i -d _concurrent| -if _concurrent/outfile-05660c60-e9ce-4b50-8e09-31392aed6957- -of count14.txt -k -status files -a -w _concurrent/outfile-05660c60-e9ce-4b50-8e09-31392aed6957- '-if' is not an internal or external command, operable program or batch file. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tianyu491433909 at 163.com Thu Jun 2 04:49:10 2011 From: tianyu491433909 at 163.com (tianyu491433909) Date: Thu, 2 Jun 2011 17:49:10 +0800 (CST) Subject: [Swift-user] a windows nfs error Message-ID: <799a110f.1e4ab.1304fc1ac52.Coremail.tianyu491433909@163.com> I have problems using the windows nfs? \\172.16.254.50\tmp\modisdata_s/\\172.16.254.50\tmp\modisdata_s\h03v10.tif the file address has two repeat folder -------------- next part -------------- An HTML attachment was scrubbed... URL: From wilde at mcs.anl.gov Thu Jun 2 10:44:06 2011 From: wilde at mcs.anl.gov (Michael Wilde) Date: Thu, 2 Jun 2011 10:44:06 -0500 (CDT) Subject: [Swift-user] a cscript.exe error In-Reply-To: <42b199.1194d.1304f7ca0cc.Coremail.tianyu491433909@163.com> Message-ID: <338402298.135672.1307029446298.JavaMail.root@zimbra.anl.gov> Hi, I wanted to let you know that we may not be able to provide help on the Swift Windows environment very quickly - it make take a week or more to find someone with time to help on this. Regards, - Mike ----- Original Message ----- I have problems using the windows environment? cscript.exe //Nologo shared\_swiftwrap.vbs wc-52333yak -jobdir 5 -scratch -e C:/swift/usr/local/wbin/wc.exe -out count14.txt -err stderr.txt -i -d _concurrent| -if _concurrent/outfile-05660c60-e9ce-4b50-8e09-31392aed6957- -of count14.txt -k -status files -a -w _concurrent/outfile-05660c60-e9ce-4b50-8e09-31392aed6957- '-if' is not an internal or external command, operable program or batch file. _______________________________________________ Swift-user mailing list Swift-user at ci.uchicago.edu http://mail.ci.uchicago.edu/mailman/listinfo/swift-user -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory -------------- next part -------------- An HTML attachment was scrubbed... URL: From iraicu at cs.iit.edu Sun Jun 5 09:44:59 2011 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Sun, 05 Jun 2011 09:44:59 -0500 Subject: [Swift-user] ACM ScienceCloud2011 is 3 days away -- final program is online Message-ID: <4DEB966B.1030603@cs.iit.edu> Dear all, We just wanted to bring to your attention that ScienceCloud2011 is only 3 days away, and will take place on Wednesday, June 8th, from 9AM to 5PM, in San Jose California, co-located with HPDC 2011, and the FCRC 2011 conferences. We have posted the final program online at http://www.cs.iit.edu/~iraicu/ScienceCloud2011/program.htm; note that we already posted the paper PDF files, and we will post the slides in the coming days. We have an exciting keynote given by Dr. Ion Stoica on "Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center". Dr. Stoica is an Associate Professor in the EECS Department at University of California at Berkeley, where he does research on cloud computing and networked computer systems. He is also the co-founder of Conviva, a startup to commercialize technologies for large scale video distribution. You will find 8 exciting talks that span industry (Yahoo, Microsoft), government labs (LBNL, ANL, JPL), and academia (UCBerkeley, USC, UWashington, CalTech, UVirginia, UCSB). The list of presentations are: 1. The Case for Being Lazy: How to Leverage Lazy Evaluation in MapReduce 2. Debunking some Common Misconceptions of Science in the Cloud 3. Experiences Using Cloud Computing for A Scientific Workflow Application 4. Cumulus: Open Source Storage Cloud for Science 5. Adaptive Rate Stream Processing for Smart Grid Applications on Clouds 6. An Automated Approach to Cloud Storage Service Selection 7. Magellan: Experiences from a Science Cloud 8. Neptune: A Domain Specific Language for Deploying HPC Software on Cloud Platforms For more information about the workshop, please visit http://www.cs.iit.edu/~iraicu/ScienceCloud2011/. We look forward to seeing you at the workshop! Regards, Ioan Raicu, Pete Beckman, Ian Foster & Yogesh Simmhan ScienceCloud2011 Co-Chairs & Program Chair http://www.cs.iit.edu/~iraicu/ScienceCloud2011/ -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ ================================================================= ================================================================= From frank.wang04 at gmail.com Fri Jun 10 01:10:12 2011 From: frank.wang04 at gmail.com (Weiyang Wang) Date: Fri, 10 Jun 2011 14:10:12 +0800 Subject: [Swift-user] [Swift]Cancelling Job Issues Message-ID: Hello, In recent several runs using SWIFT I've notice a frequent problem (which does not occur each time) when SWIFT scheduler is canceling the last active job. I'm using PADS and SWIFT-0.92.1, previous stable also share this problem. I submitted hundreds to thousands of jobs at a time, the job are finished well until it goes to the very last step -- the time point I have no "Selecting Site" "Submitted" or "Stage In" status jobs, so SWIFT's status the the screen is like "Active: 24 Stage Out: xxx Finished Successfully: xxx Failed But Can Retry: xxx" at this step the last job is cancelled despite there's still remaining Active jobs. Weiyang -------------- next part -------------- An HTML attachment was scrubbed... URL: From wilde at mcs.anl.gov Fri Jun 10 08:18:15 2011 From: wilde at mcs.anl.gov (Michael Wilde) Date: Fri, 10 Jun 2011 08:18:15 -0500 (CDT) Subject: [Swift-user] [Swift]Cancelling Job Issues In-Reply-To: Message-ID: <1875212092.25446.1307711895225.JavaMail.root@zimbra.anl.gov> Wei, is this happening with the coaster provider? If so its possible that coasters is canceling workers when their walltime remaining is not large enough to execute any ready job (each of which has either an explicit maxwalltime of a default of I think 10 minutes). I don't *think* its canceling jobs - I suspect its cancelling the *worker* job (i.e., the "pilot" job). Please send us a pointer to your script, configuration files, and log files. Also please be sure to save stdout and stderr to a file for each run. Thanks, - Mike ----- Original Message ----- Hello, In recent several runs using SWIFT I've notice a frequent problem (which does not occur each time) when SWIFT scheduler is canceling the last active job. I'm using PADS and SWIFT-0.92.1, previous stable also share this problem. I submitted hundreds to thousands of jobs at a time, the job are finished well until it goes to the very last step -- the time point I have no "Selecting Site" "Submitted" or "Stage In" status jobs, so SWIFT's status the the screen is like "Active: 24 Stage Out: xxx Finished Successfully: xxx Failed But Can Retry: xxx" at this step the last job is cancelled despite there's still remaining Active jobs. Weiyang _______________________________________________ Swift-user mailing list Swift-user at ci.uchicago.edu http://mail.ci.uchicago.edu/mailman/listinfo/swift-user -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory -------------- next part -------------- An HTML attachment was scrubbed... URL: From ketancmaheshwari at gmail.com Tue Jun 21 16:59:39 2011 From: ketancmaheshwari at gmail.com (Ketan Maheshwari) Date: Tue, 21 Jun 2011 16:59:39 -0500 Subject: [Swift-user] mapping a 2 dim array to a directory Message-ID: Hello, I am trying to get a 2 dim array to map to a directory with a given pattern as follows: The directory contains files with names as follows: a_00 a_01 . . a_99 b_00 b_01 . . . z_99 The mapped variable amap[i][j] should be mapped in such a way that i should map to the prefix and j should map to the suffix part of the file. For example: amap[2][10] should return the file c_10. In addition the said directory is an output directory and does not exist yet. A one dimensional array mapping exists for this which is as follows: file dat_files[] < simple_mapper; padding = 3, location = "output", prefix = @strcat( str_root, "_" ), suffix = ".dat" >; I looked into external mappers but could not figure out how exactly to go for it. Any clues? Regards, Ketan -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidkelly999 at gmail.com Tue Jun 21 18:18:42 2011 From: davidkelly999 at gmail.com (David Kelly) Date: Tue, 21 Jun 2011 18:18:42 -0500 Subject: [Swift-user] mapping a 2 dim array to a directory In-Reply-To: References: Message-ID: Hey Ketan, I ran into something like this recently. I used an external mapper and did something like this: type file; file myfiles[][] ; foreach farray,i in myfiles { foreach myfile,j in farray { tracef("myfiles[%i][%i] is %s\n", i, j, @filename(myfiles[i][j])); } } The mapper just has to know something about how you want your data organized. Here's a simple mapper that should work for your example: #!/usr/bin/perl -w use strict; my @alphabet = ("a" .. "z"); foreach my $count(0..scalar(@alphabet)-1) { my @files = `find . -name \'$alphabet[$count]*\'`; foreach my $filecount(0..scalar(@files)-1) { print "[$count][$filecount] $files[$filecount]"; } } So basically this just uses the find command to print something like this: [0][0] a_00 [0][1] a_01 [0][2] a_02 [1][0] b_00 I think something like that should do the trick. David On Tue, Jun 21, 2011 at 4:59 PM, Ketan Maheshwari < ketancmaheshwari at gmail.com> wrote: > Hello, > > I am trying to get a 2 dim array to map to a directory with a given pattern > as follows: > > The directory contains files with names as follows: > > a_00 > a_01 > . > . > a_99 > b_00 > b_01 > . > . > . > z_99 > > > The mapped variable amap[i][j] should be mapped in such a way that i should > map to the prefix and j should map to the suffix part of the file. > > For example: amap[2][10] should return the file c_10. > > In addition the said directory is an output directory and does not exist > yet. > > A one dimensional array mapping exists for this which is as follows: > > file dat_files[] < simple_mapper; > padding = 3, > location = "output", > prefix = @strcat( str_root, "_" ), > suffix = ".dat" >; > > I looked into external mappers but could not figure out how exactly to go > for it. > > Any clues? > > > > Regards, > Ketan > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ketancmaheshwari at gmail.com Tue Jun 21 19:36:05 2011 From: ketancmaheshwari at gmail.com (Ketan Maheshwari) Date: Tue, 21 Jun 2011 19:36:05 -0500 Subject: [Swift-user] mapping a 2 dim array to a directory In-Reply-To: References: Message-ID: That looks cool David. Thanks a lot! Ketan On Tue, Jun 21, 2011 at 6:18 PM, David Kelly wrote: > Hey Ketan, > > I ran into something like this recently. I used an external mapper and did > something like this: > > type file; > file myfiles[][] ; > > foreach farray,i in myfiles { > foreach myfile,j in farray { > tracef("myfiles[%i][%i] is %s\n", i, j, @filename(myfiles[i][j])); > } > } > > The mapper just has to know something about how you want your data > organized. Here's a simple mapper that should work for your example: > > #!/usr/bin/perl -w > use strict; > > my @alphabet = ("a" .. "z"); > > foreach my $count(0..scalar(@alphabet)-1) { > my @files = `find . -name \'$alphabet[$count]*\'`; > foreach my $filecount(0..scalar(@files)-1) { > print "[$count][$filecount] $files[$filecount]"; > } > } > > So basically this just uses the find command to print something like this: > > [0][0] a_00 > [0][1] a_01 > [0][2] a_02 > [1][0] b_00 > > I think something like that should do the trick. > > David > > On Tue, Jun 21, 2011 at 4:59 PM, Ketan Maheshwari < > ketancmaheshwari at gmail.com> wrote: > >> Hello, >> >> I am trying to get a 2 dim array to map to a directory with a given >> pattern as follows: >> >> The directory contains files with names as follows: >> >> a_00 >> a_01 >> . >> . >> a_99 >> b_00 >> b_01 >> . >> . >> . >> z_99 >> >> >> The mapped variable amap[i][j] should be mapped in such a way that i >> should map to the prefix and j should map to the suffix part of the file. >> >> For example: amap[2][10] should return the file c_10. >> >> In addition the said directory is an output directory and does not exist >> yet. >> >> A one dimensional array mapping exists for this which is as follows: >> >> file dat_files[] < simple_mapper; >> padding = 3, >> location = "output", >> prefix = @strcat( str_root, "_" ), >> suffix = ".dat" >; >> >> I looked into external mappers but could not figure out how exactly to go >> for it. >> >> Any clues? >> >> >> >> Regards, >> Ketan >> >> >> _______________________________________________ >> Swift-user mailing list >> Swift-user at ci.uchicago.edu >> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: