From hategan at mcs.anl.gov Thu Aug 7 20:56:15 2014 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 7 Aug 2014 18:56:15 -0700 Subject: [Swift-devel] direct mode Message-ID: <1407462975.20606.13.camel@echo> Hi, This is a report on the direct mode situation. I wanted to send this before the meeting tomorrow, so we can discuss the issues. It is fairly easy to implement a manual direct mode, where one would use direct:// in the mapper file specification. This requires a modification of filename()/filenames(), as well as skipping the staging of files with direct://. It seems more difficult to implement a general direct staging mode. Here's why: - since staging modes are now per-site, the exact result of filename() can only be properly assessed after site selection, which happens after filename() is invoked. This can be addressed by having filename() return an object that gets resolved from execute2(), and I have some uncommitted code for this. - the notion of direct cannot readily be used in all places. Some parts of swift still assume certain paths for certain things, such as the wrapper log, stdout and stderr files (if not specified by the user), the location of _swiftwrap (expected to be in the job directory). This would require either a modification to all places that expect such default locations, or a fallback staging mode (i.e. direct as much as you can, and then proxy or file for things that can't be direct). Mihael From wilde at anl.gov Sat Aug 9 13:54:39 2014 From: wilde at anl.gov (Michael Wilde) Date: Sat, 9 Aug 2014 13:54:39 -0500 Subject: [Swift-devel] direct mode In-Reply-To: <1407462975.20606.13.camel@echo> References: <1407462975.20606.13.camel@echo> Message-ID: <53E66E6F.8080307@anl.gov> On 8/7/14, 8:56 PM, Mihael Hategan wrote: > ... > It is fairly easy to implement a manual direct mode, where one would use > direct:// in the mapper file specification. This requires a modification > of filename()/filenames(), as well as skipping the staging of files with > direct://. Following up on this discussion from yesterday's meeting: On re-thinking this: I can see the value of the direct:// prefix, especially if its prepended to the filename using a function, eg: file f ; This is preferable to the current practice of using strings instead of files for direct access, as it preserves automatic dependency processing. This would be used when some files (e.g., large files that should be moved to a site once per run) need to be handled differently than the site's default data management strategy. This could generalize by giving each data management stratetegy a name; each site would have a default strategy declared in the site entry, and each file could get a different strategy by specifying a prefix, either in the filename URL protocol field or in the URL via a function like "direct( )", above. Thoughts? - Mike From hategan at mcs.anl.gov Sat Aug 9 15:45:18 2014 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sat, 9 Aug 2014 13:45:18 -0700 Subject: [Swift-devel] direct mode In-Reply-To: <53E66E6F.8080307@anl.gov> References: <1407462975.20606.13.camel@echo> <53E66E6F.8080307@anl.gov> Message-ID: <1407617118.24438.9.camel@echo> On Sat, 2014-08-09 at 13:54 -0500, Michael Wilde wrote: > On 8/7/14, 8:56 PM, Mihael Hategan wrote: > > ... > > It is fairly easy to implement a manual direct mode, where one would use > > direct:// in the mapper file specification. This requires a modification > > of filename()/filenames(), as well as skipping the staging of files with > > direct://. > Following up on this discussion from yesterday's meeting: > > On re-thinking this: I can see the value of the direct:// prefix, > especially if its prepended to the filename using a function, eg: > > file f ; It could just be file f <"direct:///etc/group">; It's in line with what is currently supported. For example: file f <"gsiftp://...">; or file f <"http://google.com/index.html">; > > This is preferable to the current practice of using strings instead of > files for direct access, as it preserves automatic dependency processing. I'm not sure what you mean. The dependency processing is independent of mapping and it happens at the variable level (i.e. the f variable). > > This would be used when some files (e.g., large files that should be > moved to a site once per run) need to be handled differently than the > site's default data management strategy. Actually that is what happens in swift staging mode by default. Input files are only moved once to the site, and then a link is created in the job directory. > > This could generalize by giving each data management stratetegy a name; > each site would have a default strategy declared in the site entry, and > each file could get a different strategy by specifying a prefix, either > in the filename URL protocol field or in the URL via a function like > "direct( )", above. To a large extend this is the current behavior. Sites have a staging mode, and that mode refers to how files on the swift client side are staged. When a path is specified in the mapper, it defaults to referring to a client-local file. This can be overridden using a full URL (e.g. gsiftp://site.com/...). > > Thoughts? We might need to refine this further. I'm not sure I fully understand what you are saying. In any event, my intent was to provide both a direct staging mode and a direct:// override in the URL scheme of individual files. Mihael From wilde at anl.gov Sat Aug 9 15:53:26 2014 From: wilde at anl.gov (Michael Wilde) Date: Sat, 9 Aug 2014 15:53:26 -0500 Subject: [Swift-devel] direct mode In-Reply-To: <1407617118.24438.9.camel@echo> References: <1407462975.20606.13.camel@echo> <53E66E6F.8080307@anl.gov> <1407617118.24438.9.camel@echo> Message-ID: <53E68A46.3010506@anl.gov> On 8/9/14, 3:45 PM, Mihael Hategan wrote: > On Sat, 2014-08-09 at 13:54 -0500, Michael Wilde wrote: >> On 8/7/14, 8:56 PM, Mihael Hategan wrote: >>> ... >>> It is fairly easy to implement a manual direct mode, where one would use >>> direct:// in the mapper file specification. This requires a modification >>> of filename()/filenames(), as well as skipping the staging of files with >>> direct://. >> Following up on this discussion from yesterday's meeting: >> >> On re-thinking this: I can see the value of the direct:// prefix, >> especially if its prepended to the filename using a function, eg: >> >> file f ; > It could just be > file f <"direct:///etc/group">; > > It's in line with what is currently supported. For example: > > file f <"gsiftp://...">; > or > file f <"http://google.com/index.html">; When the filename is in a var, e.g., fname, direct(fname) reads cleaner than strcat("direct://",fname). But not by much, so perhaps set that idea aside. >> This is preferable to the current practice of using strings instead of >> files for direct access, as it preserves automatic dependency processing. > I'm not sure what you mean. The dependency processing is independent of > mapping and it happens at the variable level (i.e. the f variable). Yes, but when passing string filenames, the return is not a filename, it is typically a dummy var of type external, and the outfile filename is passed as an input string. So the dependency processing doesnt help you when passing output filenames as strings. >> This would be used when some files (e.g., large files that should be >> moved to a site once per run) need to be handled differently than the >> site's default data management strategy. > Actually that is what happens in swift staging mode by default. Input > files are only moved once to the site, and then a link is created in the > job directory. But not for provider staging, as far as I understand. > In any event, my intent was to provide both a direct staging mode and a > direct:// override in the URL scheme of individual files. That sounds good. - Mike -- Michael Wilde Mathematics and Computer Science Computation Institute Argonne National Laboratory The University of Chicago From hategan at mcs.anl.gov Sat Aug 9 16:02:53 2014 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sat, 9 Aug 2014 14:02:53 -0700 Subject: [Swift-devel] direct mode In-Reply-To: <53E68A46.3010506@anl.gov> References: <1407462975.20606.13.camel@echo> <53E66E6F.8080307@anl.gov> <1407617118.24438.9.camel@echo> <53E68A46.3010506@anl.gov> Message-ID: <1407618173.24438.17.camel@echo> On Sat, 2014-08-09 at 15:53 -0500, Michael Wilde wrote: > On 8/9/14, 3:45 PM, Mihael Hategan wrote: > >> > >> file f ; > > It could just be > > file f <"direct:///etc/group">; > > > > It's in line with what is currently supported. For example: > > > > file f <"gsiftp://...">; > > or > > file f <"http://google.com/index.html">; > When the filename is in a var, e.g., fname, direct(fname) reads cleaner > than strcat("direct://",fname). I see. Yeah, but you can: - have "direct://" in fname to begin with - say "direct://" + fname. We've had the concatenation operator for a while. > But not by much, so perhaps set that idea aside. > >> This is preferable to the current practice of using strings instead of > >> files for direct access, as it preserves automatic dependency processing. > > I'm not sure what you mean. The dependency processing is independent of > > mapping and it happens at the variable level (i.e. the f variable). > Yes, but when passing string filenames, the return is not a filename, it > is typically a dummy var of type external, and the outfile filename is > passed as an input string. So the dependency processing doesnt help you > when passing output filenames as strings. Well, right. Though I suspect that passing string file names was a hack intended to emulate what direct:// would do. So, yes, this would be a proper implementation of direct mode with dependency tracking. > >> This would be used when some files (e.g., large files that should be > >> moved to a site once per run) need to be handled differently than the > >> site's default data management strategy. > > Actually that is what happens in swift staging mode by default. Input > > files are only moved once to the site, and then a link is created in the > > job directory. > But not for provider staging, as far as I understand. Right. The lack of caching in provider staging is, I believe, a separate issue that needs to be solved by adding caching to provider staging (well, coasters). But the issue of optimizing staging in general is something that we probably need to discuss further. My fear is that if we move the discussion in that direction, we might not get to a direct mode very soon. Mihael From yadudoc1729 at gmail.com Mon Aug 11 15:34:44 2014 From: yadudoc1729 at gmail.com (Yadu Nand) Date: Mon, 11 Aug 2014 15:34:44 -0500 Subject: [Swift-devel] Swift staging method names Message-ID: Hi Everyone, In the current conventions for staging, we are calling what used to be proxy staging method as local, which I think is counter-intuitive, when local staging is actually staging to remote sites. Staging method "swift" does not give any idea about what it actually does either. I'm pasting the staging methods table from the documentation below. Staging MethodDescription swift This method instructs Swift to use a filesystem provider to direct all necessary staging operations from the Swift client-side to the cluster head node. If this method is used, the workDirectory must point to a head node path that is on a shared file system accessible by the compute nodes. wrapper File staging is done by the Swift application wrapper local Used to indicate that files should be staged in/out from/to the site on which Swift is running. In the case of Swift Coasters, the system proxies the tranfers between client side and compute nodes through the Coaster Service. service-local This method instructs the execution mechanism provider to stage input and output files from the remote site where the execution service is located. For example, if a Coaster Service is started on the login node of a cluster, the Coaster Service will perform the staging from a file system on the login node to the compute node and back. shared-fs This method is used by Coasters to implement a simple staging mechanism in which files are accessed using a shared filesystem that is accessible by compute nodes ?Thanks,? Yadu Nand B -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.g.armstrong at gmail.com Mon Aug 11 16:09:10 2014 From: tim.g.armstrong at gmail.com (Tim Armstrong) Date: Mon, 11 Aug 2014 16:09:10 -0500 Subject: [Swift-devel] Swift staging method names In-Reply-To: References: Message-ID: I agree, I don't think the names give a good idea of what they're about. I also feel like it's difficult to explain some of these without diagrams. - Tim On Mon, Aug 11, 2014 at 3:34 PM, Yadu Nand wrote: > Hi Everyone, > > In the current conventions for staging, we are calling what used to be > proxy staging method as local, which I think is counter-intuitive, when > local staging is actually staging to remote sites. > > Staging method "swift" does not give any idea about what it actually does > either. > > I'm pasting the staging methods table from the documentation below. > > Staging MethodDescription > > swift > > This method instructs Swift to use a filesystem provider to direct all > necessary staging operations from the Swift client-side to the cluster head > node. If this method is used, the workDirectory must point to a head node > path that is on a shared file system accessible by the compute nodes. > > wrapper > > File staging is done by the Swift application wrapper > > local > > Used to indicate that files should be staged in/out from/to the site on > which Swift is running. In the case of Swift Coasters, the system proxies > the tranfers between client side and compute nodes through the Coaster > Service. > > service-local > > This method instructs the execution mechanism provider to stage input and > output files from the remote site where the execution service is located. > For example, if a Coaster Service is started on the login node of a > cluster, the Coaster Service will perform the staging from a file system on > the login node to the compute node and back. > > shared-fs > > This method is used by Coasters to implement a simple staging mechanism in > which files are accessed using a shared filesystem that is accessible by > compute nodes > > > ?Thanks,? > Yadu Nand B > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Mon Aug 11 16:46:44 2014 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 11 Aug 2014 14:46:44 -0700 Subject: [Swift-devel] Swift staging method names In-Reply-To: References: Message-ID: <1407793604.30237.6.camel@echo> Right. I don't think the names will be able to fully convey what they do, and we'll need documentation for that, but I think we can do better at making the names more representative of what they... represent. So far we have the following things: - swift does staging in swift-int using some filesystem provider - swift tells the provider that it needs files staged: - from the client side - from the head node - from some directory on the compute node (as an aside, I'm not sure these should necessarily be schemes in the URL, but we could instead have file://localhost/foo.txt, file://#headnode/foo.txt, file://#computenode/foo.txt). - a special version of _swiftwrap does the staging - swift passes direct paths to the app as much as possible and tells the provider that it needs some files staged for the rest - *MAYBE* swift passes direct paths to the app as much as possible and (does staging of|tells _swiftwrap to stage) the rest Mihael On Mon, 2014-08-11 at 16:09 -0500, Tim Armstrong wrote: > I agree, I don't think the names give a good idea of what they're about. I > also feel like it's difficult to explain some of these without diagrams. > > - Tim > > > On Mon, Aug 11, 2014 at 3:34 PM, Yadu Nand wrote: > > > Hi Everyone, > > > > In the current conventions for staging, we are calling what used to be > > proxy staging method as local, which I think is counter-intuitive, when > > local staging is actually staging to remote sites. > > > > Staging method "swift" does not give any idea about what it actually does > > either. > > > > I'm pasting the staging methods table from the documentation below. > > > > Staging MethodDescription > > > > swift > > > > This method instructs Swift to use a filesystem provider to direct all > > necessary staging operations from the Swift client-side to the cluster head > > node. If this method is used, the workDirectory must point to a head node > > path that is on a shared file system accessible by the compute nodes. > > > > wrapper > > > > File staging is done by the Swift application wrapper > > > > local > > > > Used to indicate that files should be staged in/out from/to the site on > > which Swift is running. In the case of Swift Coasters, the system proxies > > the tranfers between client side and compute nodes through the Coaster > > Service. > > > > service-local > > > > This method instructs the execution mechanism provider to stage input and > > output files from the remote site where the execution service is located. > > For example, if a Coaster Service is started on the login node of a > > cluster, the Coaster Service will perform the staging from a file system on > > the login node to the compute node and back. > > > > shared-fs > > > > This method is used by Coasters to implement a simple staging mechanism in > > which files are accessed using a shared filesystem that is accessible by > > compute nodes > > > > > > ?Thanks,? > > Yadu Nand B > > > > > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel From wilde at anl.gov Fri Aug 15 09:33:20 2014 From: wilde at anl.gov (Michael Wilde) Date: Fri, 15 Aug 2014 09:33:20 -0500 Subject: [Swift-devel] expression typing Message-ID: <53EE1A30.8010801@anl.gov> A participant in yesterday's ATPESC tutorial was surprised to see that division of two integers yields a float type. We should discuss this, as well as unification of the type coercion rules of both Swift/K and Swift/T, and document what the situation is (as well as whether we plan to change it). - Mike -- Michael Wilde Mathematics and Computer Science Computation Institute Argonne National Laboratory The University of Chicago From wozniak at mcs.anl.gov Fri Aug 15 09:39:15 2014 From: wozniak at mcs.anl.gov (Justin M Wozniak) Date: Fri, 15 Aug 2014 09:39:15 -0500 Subject: [Swift-devel] expression typing In-Reply-To: <53EE1A30.8010801@anl.gov> References: <53EE1A30.8010801@anl.gov> Message-ID: <53EE1B93.9040303@mcs.anl.gov> Both Swifts use / for floating point division and %/ for integer division, both are in the docs. On 08/15/2014 09:33 AM, Michael Wilde wrote: > A participant in yesterday's ATPESC tutorial was surprised to see that > division of two integers yields a float type. > > We should discuss this, as well as unification of the type coercion > rules of both Swift/K and Swift/T, and document what the situation is > (as well as whether we plan to change it). > > - Mike > -- Justin M Wozniak From tim.g.armstrong at gmail.com Fri Aug 15 10:50:02 2014 From: tim.g.armstrong at gmail.com (Tim Armstrong) Date: Fri, 15 Aug 2014 10:50:02 -0500 Subject: [Swift-devel] expression typing In-Reply-To: <53EE1B93.9040303@mcs.anl.gov> References: <53EE1A30.8010801@anl.gov> <53EE1B93.9040303@mcs.anl.gov> Message-ID: Yes, it's been like this for a long time (it's the same in Swift/K). Generally I think it's best to not diverge from convention unnecessarily, but the traditional C behaviour is so error prone - you have two related but distinct mathematical operations being denoted with the same symbol. I've seen the mistake of doing integer division instead of floating point division all the time from beginner programmers and even with some frequency in code from experienced programmers who understand the behaviour but had a mental lapse (myself included). Neither implementation does much in the way of automatic type coercion. In general I have avoided it since there's a high probability of unintended consequences, except when a) there's no chance of information loss and b) they're of the same general category of types, e.g. numeric. The only differences I could find were: 1. Swift/T always casts integer literals to floats provided they're exactly representable, Swift/K only seems to when they're arguments to operators. 2. Swift/K integers are 32-bits, which means that in context where they might be casted, they can always be cast to (double-precision) floats without information loss. In Swift/T, they're 64-bit, so you can lose precision casting to double - Swift/T won't coerce integers to floats unless it's a literal where it knows the value. 3. Swift/K will coerce numeric types to strings in very limited situations (when applying the + operator to a string and a numeric type). Swift/T doesn't do this. I've created a ticket for the last feature in Swift/T: https://code.google.com/p/exm-issues/issues/detail?id=733 - Tim On Fri, Aug 15, 2014 at 9:39 AM, Justin M Wozniak wrote: > > Both Swifts use / for floating point division and %/ for integer > division, both are in the docs. > > On 08/15/2014 09:33 AM, Michael Wilde wrote: > > A participant in yesterday's ATPESC tutorial was surprised to see that > > division of two integers yields a float type. > > > > We should discuss this, as well as unification of the type coercion > > rules of both Swift/K and Swift/T, and document what the situation is > > (as well as whether we plan to change it). > > > > - Mike > > > > > -- > Justin M Wozniak > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.g.armstrong at gmail.com Fri Aug 15 16:56:18 2014 From: tim.g.armstrong at gmail.com (Tim Armstrong) Date: Fri, 15 Aug 2014 16:56:18 -0500 Subject: [Swift-devel] Configuring standalone active coasters service Message-ID: I'm looking for some information about configuring coasters services. For Swift/T + Coasters I want to have a standalone coaster service that can actively provision new workers. Up until now I've been using passive mode via start-coaster-service. I've been poking around documentation and examples to try to understand how I would configure the active service. >From what I can tell, it would involve: 1. Running coaster-service to start up the service. 2. Configure the service with site information Part 2 is giving me trouble. Is there a way to do this configuration through the coaster-service invocation, or do I have to send the config information over the wire from the client? If it's the second, is there some way I can work out what setting can/must be provided? - Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Fri Aug 15 18:20:53 2014 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Fri, 15 Aug 2014 16:20:53 -0700 Subject: [Swift-devel] Configuring standalone active coasters service In-Reply-To: References: Message-ID: <1408144853.4473.9.camel@echo> On Fri, 2014-08-15 at 16:56 -0500, Tim Armstrong wrote: > I'm looking for some information about configuring coasters services. For > Swift/T + Coasters I want to have a standalone coaster service that can > actively provision new workers. Up until now I've been using passive mode > via start-coaster-service. I've been poking around documentation and > examples to try to understand how I would configure the active service. > From what I can tell, it would involve: > > 1. Running coaster-service to start up the service. > 2. Configure the service with site information > > Part 2 is giving me trouble. Is there a way to do this configuration > through the coaster-service invocation, or do I have to send the config > information over the wire from the client? If it's the second, is there > some way I can work out what setting can/must be provided? It's over the wire from the client right now. There is a bug for adding a startup configuration scheme (which would be shared by all clients). There coaster options were supposed to be in the swift user guide, but I can't find them in the trunk version. Some of them are there for 0.94: http://swift-lang.org/guides/release-0.94/userguide/userguide.html#_coasters_how_to Some old coaster document lists them without much explanation, and they might not be up to date: http://wiki.cogkit.org/wiki/Coaster_Protocol#Configuration_Options This situation needs serious fixing. org.globus.cog.abstraction.coaster.service.job.manager.Settings may also be helpful. It has some descriptions of the various fields. Mihael From hategan at mcs.anl.gov Fri Aug 15 22:34:14 2014 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Fri, 15 Aug 2014 20:34:14 -0700 Subject: [Swift-devel] direct Message-ID: <1408160054.6861.0.camel@echo> Hi, Sorry for the delay. Direct is now in. Use with "staging: direct" in swift.conf. Only works with local and coaster providers. Mihael From tim.g.armstrong at gmail.com Sat Aug 16 22:10:07 2014 From: tim.g.armstrong at gmail.com (Tim Armstrong) Date: Sat, 16 Aug 2014 22:10:07 -0500 Subject: [Swift-devel] Configuring standalone active coasters service In-Reply-To: <1408157146.6212.5.camel@echo> References: <1408144853.4473.9.camel@echo> <1408157146.6212.5.camel@echo> Message-ID: Thanks, that helps. CC'ing list again since I dropped it earlier. - Tim On Fri, Aug 15, 2014 at 9:45 PM, Mihael Hategan wrote: > On Fri, 2014-08-15 at 19:32 -0500, Tim Armstrong wrote: > > Just checking my understanding... Do the coaster options you send over > the > > wire correspond directly to the options {} dict here: > > > http://swift-lang.org/guides/trunk/userguide/userguide.html#_configuration_file_structure > > > > I.e. options in: > > > > execution { > > type: "coaster" > > jobManager: "local:local" > > options { > > maxJobs: 1 > > tasksPerNode: 2 > > workerLoggingLevel: TRACE > > } > > } > > Yes, except the jobOptions are also flattened and sent along with the > other ones. > > > > > > > Also, how do you set parameters specific to the jobManager, e.g. if > you're > > submitting through pbs:local, how do you set the PBS queue? > > Those are attributes. So everything that isn't specifically a coaster > setting will be passed to the job. > > Mihael > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Tue Aug 19 00:49:11 2014 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 18 Aug 2014 22:49:11 -0700 Subject: [Swift-devel] computer issues Message-ID: <1408427351.6684.3.camel@echo> Hi, So my laptop crashed a bunch of times today failing to write to disk with I/O errors and such. I did a full image of the disk and that worked with no errors. SMART isn't showing any problems either. So I'm going to blame this on the kernel. I switched to a previous version for now and it's been stable for a few minutes. Anyway, the jury isn't fully out yet, so just a heads up if I seem to have disappeared. Mihael From ketan at mcs.anl.gov Fri Aug 22 13:41:42 2014 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Fri, 22 Aug 2014 13:41:42 -0500 Subject: [Swift-devel] error building swift trunk on bluegene Message-ID: Hi, I am getting this error trying to build Swift trunk on BG Vesta: compile: [echo] [util]: COMPILE [mkdir] Created dir: /gpfs/vesta-home/ketan/swift-devel/cog/modules/util/build [javac] Compiling 56 source files to /gpfs/vesta-home/ketan/swift-devel/cog/modules/util/build [javac] /gpfs/vesta-home/ketan/swift-devel/cog/modules/util/src/org/globus/cog/util/concurrent/FileLock.java:91: cannot find symbol [javac] symbol : class VMManagement [javac] location: package sun.management [javac] sun.management.VMManagement mgmt = (sun.management.VMManagement) jvm.get(runtime); [javac] ^ [javac] /gpfs/vesta-home/ketan/swift-devel/cog/modules/util/src/org/globus/cog/util/concurrent/FileLock.java:91: cannot find symbol [javac] symbol : class VMManagement [javac] location: package sun.management [javac] sun.management.VMManagement mgmt = (sun.management.VMManagement) jvm.get(runtime); [javac] ^ [javac] Note: Some input files use or override a deprecated API. [javac] Note: Recompile with -Xlint:deprecation for details. [javac] Note: Some input files use unchecked or unsafe operations. [javac] Note: Recompile with -Xlint:unchecked for details. [javac] 2 errors Doe this means I need sun java on Vesta? Does anyone know if Sun java is available for PPC64 architecture? Thanks, Ketan -------------- next part -------------- An HTML attachment was scrubbed... URL: From ketan at mcs.anl.gov Fri Aug 22 13:44:36 2014 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Fri, 22 Aug 2014 13:44:36 -0500 Subject: [Swift-devel] error building swift trunk on bluegene In-Reply-To: References: Message-ID: PS. The current java is from IBM: $ java -version java version "1.6.0" Java(TM) SE Runtime Environment (build pxp6460sr15-20131017_01(SR15)) IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux ppc64-64 jvmxp6460sr15-20131016_170922 (JIT enabled, AOT enabled) J9VM - 20131016_170922 JIT - r9_20130920_46510ifx2 GC - GA24_Java6_SR15_20131016_1337_B170922) JCL - 20131015_01 On Fri, Aug 22, 2014 at 1:41 PM, Ketan Maheshwari wrote: > Hi, > > I am getting this error trying to build Swift trunk on BG Vesta: > > compile: > [echo] [util]: COMPILE > [mkdir] Created dir: > /gpfs/vesta-home/ketan/swift-devel/cog/modules/util/build > [javac] Compiling 56 source files to > /gpfs/vesta-home/ketan/swift-devel/cog/modules/util/build > [javac] > /gpfs/vesta-home/ketan/swift-devel/cog/modules/util/src/org/globus/cog/util/concurrent/FileLock.java:91: > cannot find symbol > [javac] symbol : class VMManagement > [javac] location: package sun.management > [javac] sun.management.VMManagement mgmt = > (sun.management.VMManagement) jvm.get(runtime); > [javac] ^ > [javac] > /gpfs/vesta-home/ketan/swift-devel/cog/modules/util/src/org/globus/cog/util/concurrent/FileLock.java:91: > cannot find symbol > [javac] symbol : class VMManagement > [javac] location: package sun.management > [javac] sun.management.VMManagement mgmt = > (sun.management.VMManagement) jvm.get(runtime); > [javac] ^ > [javac] Note: Some input files use or override a deprecated API. > [javac] Note: Recompile with -Xlint:deprecation for details. > [javac] Note: Some input files use unchecked or unsafe operations. > [javac] Note: Recompile with -Xlint:unchecked for details. > [javac] 2 errors > > Doe this means I need sun java on Vesta? Does anyone know if Sun java is > available for PPC64 architecture? > > Thanks, > Ketan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wilde at anl.gov Fri Aug 22 21:04:44 2014 From: wilde at anl.gov (Michael Wilde) Date: Fri, 22 Aug 2014 21:04:44 -0500 Subject: [Swift-devel] Need new cobalt provider for BG/Q In-Reply-To: References: Message-ID: <53F7F6BC.4010802@anl.gov> We discovered today that the current cobalt provider needs to be converted from using cqsub to the BG/Q's version of Cobalt qsub. This seems like an excellent opportunity to use the new templated local scheduler provider. Mihael, can you provide instructions for how to do this? As I recall, the prior info you sent regarding this (when you first developed this feature) needed further clarification. Thanks, - Mike On 8/22/14, 1:44 PM, Ketan Maheshwari wrote: > PS. The current java is from IBM: > > $ java -version > java version "1.6.0" > Java(TM) SE Runtime Environment (build pxp6460sr15-20131017_01(SR15)) > IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux ppc64-64 > jvmxp6460sr15-20131016_170922 (JIT enabled, AOT enabled) > J9VM - 20131016_170922 > JIT - r9_20130920_46510ifx2 > GC - GA24_Java6_SR15_20131016_1337_B170922) > JCL - 20131015_01 > > > On Fri, Aug 22, 2014 at 1:41 PM, Ketan Maheshwari > wrote: > > Hi, > > I am getting this error trying to build Swift trunk on BG Vesta: > > compile: > [echo] [util]: COMPILE > [mkdir] Created dir: > /gpfs/vesta-home/ketan/swift-devel/cog/modules/util/build > [javac] Compiling 56 source files to > /gpfs/vesta-home/ketan/swift-devel/cog/modules/util/build > [javac] > /gpfs/vesta-home/ketan/swift-devel/cog/modules/util/src/org/globus/cog/util/concurrent/FileLock.java:91: > cannot find symbol > [javac] symbol : class VMManagement > [javac] location: package sun.management > [javac] sun.management.VMManagement mgmt = > (sun.management.VMManagement) jvm.get(runtime); > [javac] ^ > [javac] > /gpfs/vesta-home/ketan/swift-devel/cog/modules/util/src/org/globus/cog/util/concurrent/FileLock.java:91: > cannot find symbol > [javac] symbol : class VMManagement > [javac] location: package sun.management > [javac] sun.management.VMManagement mgmt = > (sun.management.VMManagement) jvm.get(runtime); > [javac] ^ > [javac] Note: Some input files use or override a deprecated API. > [javac] Note: Recompile with -Xlint:deprecation for details. > [javac] Note: Some input files use unchecked or unsafe operations. > [javac] Note: Recompile with -Xlint:unchecked for details. > [javac] 2 errors > > Doe this means I need sun java on Vesta? Does anyone know if Sun > java is available for PPC64 architecture? > > Thanks, > Ketan > > > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel -- Michael Wilde Mathematics and Computer Science Computation Institute Argonne National Laboratory The University of Chicago -------------- next part -------------- An HTML attachment was scrubbed... URL: From wilde at anl.gov Fri Aug 22 22:00:24 2014 From: wilde at anl.gov (Michael Wilde) Date: Fri, 22 Aug 2014 22:00:24 -0500 Subject: [Swift-devel] Need new cobalt provider for BG/Q In-Reply-To: <53F7F6BC.4010802@anl.gov> References: <53F7F6BC.4010802@anl.gov> Message-ID: <53F803C8.6080800@anl.gov> Here's a pointer to the docs for this. I should re-read before I ask for more details :) https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1245 --- Comment #4 from Mihael Hategan 2014-06-10 01:05:00 --- Cog trunk r3918 has this. There is an example in provider-localscheduler/examples as well as documentation in provider-localscheduler/shell-provider.README --- - Mike On 8/22/14, 9:04 PM, Michael Wilde wrote: > We discovered today that the current cobalt provider needs to be > converted from using cqsub to the BG/Q's version of Cobalt qsub. > > This seems like an excellent opportunity to use the new templated > local scheduler provider. > > Mihael, can you provide instructions for how to do this? > > As I recall, the prior info you sent regarding this (when you first > developed this feature) needed further clarification. > > Thanks, > > - Mike > > > > On 8/22/14, 1:44 PM, Ketan Maheshwari wrote: >> PS. The current java is from IBM: >> >> $ java -version >> java version "1.6.0" >> Java(TM) SE Runtime Environment (build pxp6460sr15-20131017_01(SR15)) >> IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux ppc64-64 >> jvmxp6460sr15-20131016_170922 (JIT enabled, AOT enabled) >> J9VM - 20131016_170922 >> JIT - r9_20130920_46510ifx2 >> GC - GA24_Java6_SR15_20131016_1337_B170922) >> JCL - 20131015_01 >> >> >> On Fri, Aug 22, 2014 at 1:41 PM, Ketan Maheshwari > > wrote: >> >> Hi, >> >> I am getting this error trying to build Swift trunk on BG Vesta: >> >> compile: >> [echo] [util]: COMPILE >> [mkdir] Created dir: >> /gpfs/vesta-home/ketan/swift-devel/cog/modules/util/build >> [javac] Compiling 56 source files to >> /gpfs/vesta-home/ketan/swift-devel/cog/modules/util/build >> [javac] >> /gpfs/vesta-home/ketan/swift-devel/cog/modules/util/src/org/globus/cog/util/concurrent/FileLock.java:91: >> cannot find symbol >> [javac] symbol : class VMManagement >> [javac] location: package sun.management >> [javac] sun.management.VMManagement mgmt = >> (sun.management.VMManagement) jvm.get(runtime); >> [javac] ^ >> [javac] >> /gpfs/vesta-home/ketan/swift-devel/cog/modules/util/src/org/globus/cog/util/concurrent/FileLock.java:91: >> cannot find symbol >> [javac] symbol : class VMManagement >> [javac] location: package sun.management >> [javac] sun.management.VMManagement mgmt = >> (sun.management.VMManagement) jvm.get(runtime); >> [javac] ^ >> [javac] Note: Some input files use or override a deprecated API. >> [javac] Note: Recompile with -Xlint:deprecation for details. >> [javac] Note: Some input files use unchecked or unsafe >> operations. >> [javac] Note: Recompile with -Xlint:unchecked for details. >> [javac] 2 errors >> >> Doe this means I need sun java on Vesta? Does anyone know if Sun >> java is available for PPC64 architecture? >> >> Thanks, >> Ketan >> >> >> >> >> _______________________________________________ >> Swift-devel mailing list >> Swift-devel at ci.uchicago.edu >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > -- > Michael Wilde > Mathematics and Computer Science Computation Institute > Argonne National Laboratory The University of Chicago > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel -- Michael Wilde Mathematics and Computer Science Computation Institute Argonne National Laboratory The University of Chicago -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Sat Aug 23 13:16:21 2014 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sat, 23 Aug 2014 11:16:21 -0700 Subject: [Swift-devel] Need new cobalt provider for BG/Q In-Reply-To: <53F803C8.6080800@anl.gov> References: <53F7F6BC.4010802@anl.gov> <53F803C8.6080800@anl.gov> Message-ID: <1408817781.9271.6.camel@echo> Right. Yadu implemented an EC2 provider using this. If there are specific questions, I'll be happy to add them to the documentation, but I think it's otherwise reasonable. However, I'm not sure if a completely new provider is the way to go. Can you provide more details about what's insufficient about cqsub? Mihael On Fri, 2014-08-22 at 22:00 -0500, Michael Wilde wrote: > Here's a pointer to the docs for this. I should re-read before I ask > for more details :) > > https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1245 > > --- Comment #4 from Mihael Hategan 2014-06-10 01:05:00 --- > Cog trunk r3918 has this. There is an example in > provider-localscheduler/examples as well as documentation in > provider-localscheduler/shell-provider.README > --- > > - Mike > > > On 8/22/14, 9:04 PM, Michael Wilde wrote: > > We discovered today that the current cobalt provider needs to be > > converted from using cqsub to the BG/Q's version of Cobalt qsub. > > > > This seems like an excellent opportunity to use the new templated > > local scheduler provider. > > > > Mihael, can you provide instructions for how to do this? > > > > As I recall, the prior info you sent regarding this (when you first > > developed this feature) needed further clarification. > > > > Thanks, > > > > - Mike > > > > > > > > On 8/22/14, 1:44 PM, Ketan Maheshwari wrote: > >> PS. The current java is from IBM: > >> > >> $ java -version > >> java version "1.6.0" > >> Java(TM) SE Runtime Environment (build pxp6460sr15-20131017_01(SR15)) > >> IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux ppc64-64 > >> jvmxp6460sr15-20131016_170922 (JIT enabled, AOT enabled) > >> J9VM - 20131016_170922 > >> JIT - r9_20130920_46510ifx2 > >> GC - GA24_Java6_SR15_20131016_1337_B170922) > >> JCL - 20131015_01 > >> > >> > >> On Fri, Aug 22, 2014 at 1:41 PM, Ketan Maheshwari >> > wrote: > >> > >> Hi, > >> > >> I am getting this error trying to build Swift trunk on BG Vesta: > >> > >> compile: > >> [echo] [util]: COMPILE > >> [mkdir] Created dir: > >> /gpfs/vesta-home/ketan/swift-devel/cog/modules/util/build > >> [javac] Compiling 56 source files to > >> /gpfs/vesta-home/ketan/swift-devel/cog/modules/util/build > >> [javac] > >> /gpfs/vesta-home/ketan/swift-devel/cog/modules/util/src/org/globus/cog/util/concurrent/FileLock.java:91: > >> cannot find symbol > >> [javac] symbol : class VMManagement > >> [javac] location: package sun.management > >> [javac] sun.management.VMManagement mgmt = > >> (sun.management.VMManagement) jvm.get(runtime); > >> [javac] ^ > >> [javac] > >> /gpfs/vesta-home/ketan/swift-devel/cog/modules/util/src/org/globus/cog/util/concurrent/FileLock.java:91: > >> cannot find symbol > >> [javac] symbol : class VMManagement > >> [javac] location: package sun.management > >> [javac] sun.management.VMManagement mgmt = > >> (sun.management.VMManagement) jvm.get(runtime); > >> [javac] ^ > >> [javac] Note: Some input files use or override a deprecated API. > >> [javac] Note: Recompile with -Xlint:deprecation for details. > >> [javac] Note: Some input files use unchecked or unsafe > >> operations. > >> [javac] Note: Recompile with -Xlint:unchecked for details. > >> [javac] 2 errors > >> > >> Doe this means I need sun java on Vesta? Does anyone know if Sun > >> java is available for PPC64 architecture? > >> > >> Thanks, > >> Ketan > >> > >> > >> > >> > >> _______________________________________________ > >> Swift-devel mailing list > >> Swift-devel at ci.uchicago.edu > >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > > -- > > Michael Wilde > > Mathematics and Computer Science Computation Institute > > Argonne National Laboratory The University of Chicago > > > > > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel From iraicu at cs.iit.edu Sat Aug 23 15:09:50 2014 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Sat, 23 Aug 2014 15:09:50 -0500 Subject: [Swift-devel] CFP: ACM MTAGS 2014 @ SC14 -- deadline extension to September 8th & keynote speakers announced Message-ID: <53F8F50E.8080200@cs.iit.edu> Call for Papers --------------------------------------------------------------------------------------- The 7th Workshop on Many-Task Computing on Clouds, Grids, and Supercomputers (MTAGS) 2014 http://datasys.cs.iit.edu/events/MTAGS14/ --------------------------------------------------------------------------------------- November 16th, 2014 New Orleans, Louisiana, USA Co-located with with IEEE/ACM International Conference for High Performance Computing, Networking, Storage and Analysis (SC14) In cooperation with ACM SIGHPC NEWS ======================================================================================= - Deadline extension to September 8th, 2014 - Confirmed keynote speakers: - Dr. Owen O'Malley, Co-founder Hortonworks Inc. - Mike Wilde, Fellow at University of Chicago and Software Architect at Argonne National Laboratory - Call for Papers: Special Issue on Many-Task Computing in the Cloud, in the IEEE Transaction on Cloud Computing -- papers due February 2015 ======================================================================================= The 7th workshop on Many-Task Computing on Clouds, Grids, and Supercomputers (MTAGS) will provide the scientific community a dedicated forum for presenting new research, development, and deployment efforts of large-scale many-task computing (MTC) applications on large scale clusters, Grids, Supercomputers, and Cloud Computing infrastructure. MTC, the theme of the workshop encompasses loosely coupled applications, which are generally composed of many tasks (both independent and dependent tasks) to achieve some larger application goal. This workshop will cover challenges that can hamper efficiency and utilization in running applications on large-scale systems, such as local resource manager scalability and granularity, efficient utilization of raw hardware, parallel file system contention and scalability, data management, I/O management, reliability at scale, and application scalability. We welcome paper submissions on all theoretical, simulations, and systems topics related to MTC, but we give special consideration to papers addressing petascale to exascale challenges. Papers will be peer-reviewed, and accepted papers will be published in the workshop proceedings as part of the ACM digital library (pending approval). The workshop will be co-located with the IEEE/ACM Supercomputing 2014 Conference in New Orleans on November 17th, 2014. For more information, please see http://datasys.cs.iit.edu/events/MTAGS14/. For more information on past workshops, please see MTAGS13, MTAGS12, MTAGS11, MTAGS10, MTAGS09, and MTAGS08. We also ran a Special Issue on Many-Task Computing in the IEEE Transactions on Parallel and Distributed Systems (TPDS) which appeared in June 2011; the proceedings can be found online at http://www.computer.org/portal/web/csdl/abs/trans/td/2011/06/ttd201106toc.htm. We, the workshop organizers, also published a highly relevant paper that defines Many-Task Computing which was published in MTAGS08, titled ?Many-Task Computing for Grids and Supercomputers?; we encourage potential authors to read this paper, and to clearly articulate in your paper submissions how your papers are related to Many-Task Computing. Topics --------------------------------------------------------------------------------------- We invite the submission of original work that is related to the topics below. The papers should be 6 pages, including all figures and references. We aim to cover topics related to Many-Task Computing on each of the three major distributed systems paradigms, Cloud Computing, Grid Computing and Supercomputing. Topics of interest include: * Compute Resource Management * Scheduling * Job execution frameworks * Local resource manager extensions * Performance evaluation of resource managers in use on large scale systems * Dynamic resource provisioning * Techniques to manage many-core resources and/or GPUs * Challenges and opportunities in running many-task workloads on HPC systems * Challenges and opportunities in running many-task workloads on Cloud Computing infrastructure * Storage architectures and implementations * Distributed file systems * Parallel file systems * Distributed meta-data management * Content distribution systems for large data * Data caching frameworks and techniques * Data management within and across data centers * Data-aware scheduling * Data-intensive computing applications * Eventual-consistency storage usage and management * Programming models and tools * Map-reduce and its generalizations * Many-task computing middleware and applications * Parallel programming frameworks * Ensemble MPI techniques and frameworks * Service-oriented science applications * Large-Scale Workflow Systems * Workflow system performance and scalability analysis * Scalability of workflow systems * Workflow infrastructure and e-Science middleware * Programming Paradigms and Models * Large-Scale Many-Task Applications * High-throughput computing (HTC) applications * Data-intensive applications * Quasi-supercomputing applications, deployments, and experiences * Performance Evaluation * Performance evaluation * Real systems * Simulations * Reliability of large systems Paper Submission and Publication --------------------------------------------------------------------------------------- Authors are invited to submit papers with unpublished, original work of not more than 6 pages of double column text using single spaced 10 point size on 8.5 x 11 inch pages, as per ACM 8.5 x 11 manuscript guidelines; document templates can be found at http://www.acm.org/sigs/publications/proceedings-templates. The final 6 page papers in PDF format must be submitted online at https://cmt.research.microsoft.com/MTAGS2014/ before the deadline of September 8th, 2014 at 11:59PM PST. Papers will be peer-reviewed, and accepted papers will be published in the workshop proceedings as part of the ACM digital library (in cooperation with SIGHPC). Notifications of the paper decisions will be sent out by September 22nd, 2014. Accepted workshop papers will be eligible for additional post-conference publication as journal articles in the IEEE Transaction on Cloud Computing, Special Issue on Many-Task Computing in the Cloud (papers will be due in February 2015, http://datasys.cs.iit.edu/events/TCC-MTC15/index.html). Submission implies the willingness of at least one of the authors to register and present the paper. For more information, please see http://datasys.cs.iit.edu/events/MTAGS14/. Important Dates --------------------------------------------------------------------------------------- * Papers Due: September 8th, 2014 * Notification of Acceptance: September 29th, 2014 * Camera Ready Papers Due: October 6th, 2014 * Workshop Date: November 16th, 2014 Committee Members --------------------------------------------------------------------------------------- Workshop Chairs * Ioan Raicu, Illinois Institute of Technology & Argonne National Laboratory * Justin Wozniak, University of Chicago & Argonne National Laboratory * Ian Foster, University of Chicago & Argonne National Laboratory * Yong Zhao, University of Electronic Science and Technology of China Steering Committee * David Abramson, Monash University, Australia * Jack Dongarra, University of Tennessee, USA * Geoffrey Fox, Indiana University, USA * Manish Parashar, Rutgers University, USA * Marc Snir, University of Illinois at Urbana Champaign, USA * Xian-He Sun, Illinois Institute of Technology, USA * Weimin Zheng, Tsinghua University, China Technical Committee * Hasan Abbasi, Oak Ridge National Labs, USA * Tim Armstrong, University of Chicago, USA * Roger Barga, Microsoft, USA * Mihai Budiu, Microsoft Research, USA * Rajkumar Buyya University of Melbourne, Australia * Kyle Chard, University of Chicago, USA * Evangelinos Constantinos, Massachusetts Institute of Technology, USA * Catalin Dumitrescu, Fermi National Labs, USA * Haryadi Gunawi, University of Chicago, USA * Indranil Gupta, University of Illinois at Urbana Champaign, USA * Alexandru Iosup, Delft University of Technology, Netherlands * Florin Isaila, Universidad Carlos III de Madrid, Spain & Argonne National Laboratory, USA * Kamil Iskra, Argonne National Laboratory, USA * Daniel S. Katz, University of Chicago, USA * Jik-Soo Kim, Kristi, Korea * Scott A. Klasky, Oak Ridge National Labs, USA * Mike Lang, Los Alamos National Laboratory, USA * Tonglin Li, Illinois Institute of Technology, USA * Chris Moretti, Princeton University, USA * David O'Hallaron, Carnegie Mellon University, USA * Marlon Pierce, Indiana University, USA * Judy Qiu, Indiana University, USA * Lavanya Ramakrishnan, Lawrence Berkeley National Laboratory, USA * Matei Ripeanu, University of British Columbia, Canada * Wei Tang, Argonne National Laboratory, USA * Edward Walker, Whitworth University, USA * Ke Wang, Illinois Institute of Technology, USA * Matthew Woitaszek, Walmart Labs, USA * Rich Wolski, University of California, Santa Barbara, USA * Zhifeng Yun, University of Houston, USA * Zhao Zhang, University of California at Berkeley, USA * Ziming Zheng, University of Chicago, USA -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Editor: IEEE TCC, Springer Cluster, Springer JoCCASA Chair: IEEE/ACM MTAGS, ACM ScienceCloud ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ LinkedIn: http://www.linkedin.com/in/ioanraicu Google: http://scholar.google.com/citations?user=jE73HYAAAAAJ ================================================================= ================================================================= From hategan at mcs.anl.gov Sat Aug 23 17:02:45 2014 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sat, 23 Aug 2014 15:02:45 -0700 Subject: [Swift-devel] error building swift trunk on bluegene In-Reply-To: References: Message-ID: <1408831365.24566.1.camel@echo> That particular issue should now be fixed in trunk. I'm able to build swift with the ibm libraries (although within eclipse, so please give this a shot in a real environment). Mihael On Fri, 2014-08-22 at 13:44 -0500, Ketan Maheshwari wrote: > PS. The current java is from IBM: > > $ java -version > java version "1.6.0" > Java(TM) SE Runtime Environment (build pxp6460sr15-20131017_01(SR15)) > IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux ppc64-64 > jvmxp6460sr15-20131016_170922 (JIT enabled, AOT enabled) > J9VM - 20131016_170922 > JIT - r9_20130920_46510ifx2 > GC - GA24_Java6_SR15_20131016_1337_B170922) > JCL - 20131015_01 > > > On Fri, Aug 22, 2014 at 1:41 PM, Ketan Maheshwari wrote: > > > Hi, > > > > I am getting this error trying to build Swift trunk on BG Vesta: > > > > compile: > > [echo] [util]: COMPILE > > [mkdir] Created dir: > > /gpfs/vesta-home/ketan/swift-devel/cog/modules/util/build > > [javac] Compiling 56 source files to > > /gpfs/vesta-home/ketan/swift-devel/cog/modules/util/build > > [javac] > > /gpfs/vesta-home/ketan/swift-devel/cog/modules/util/src/org/globus/cog/util/concurrent/FileLock.java:91: > > cannot find symbol > > [javac] symbol : class VMManagement > > [javac] location: package sun.management > > [javac] sun.management.VMManagement mgmt = > > (sun.management.VMManagement) jvm.get(runtime); > > [javac] ^ > > [javac] > > /gpfs/vesta-home/ketan/swift-devel/cog/modules/util/src/org/globus/cog/util/concurrent/FileLock.java:91: > > cannot find symbol > > [javac] symbol : class VMManagement > > [javac] location: package sun.management > > [javac] sun.management.VMManagement mgmt = > > (sun.management.VMManagement) jvm.get(runtime); > > [javac] ^ > > [javac] Note: Some input files use or override a deprecated API. > > [javac] Note: Recompile with -Xlint:deprecation for details. > > [javac] Note: Some input files use unchecked or unsafe operations. > > [javac] Note: Recompile with -Xlint:unchecked for details. > > [javac] 2 errors > > > > Doe this means I need sun java on Vesta? Does anyone know if Sun java is > > available for PPC64 architecture? > > > > Thanks, > > Ketan > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel From ketan at mcs.anl.gov Mon Aug 25 11:35:38 2014 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Mon, 25 Aug 2014 11:35:38 -0500 Subject: [Swift-devel] Need new cobalt provider for BG/Q In-Reply-To: <1408817781.9271.6.camel@echo> References: <53F7F6BC.4010802@anl.gov> <53F803C8.6080800@anl.gov> <1408817781.9271.6.camel@echo> Message-ID: Hi Mihael, I have implemented the cqsub -> qsub change manually for cobalt provider. Not yet tested as the BG machines are down for scheduled maintenance. Will keep you posted. Thanks, Ketan On Sat, Aug 23, 2014 at 1:16 PM, Mihael Hategan wrote: > Right. > > Yadu implemented an EC2 provider using this. If there are specific > questions, I'll be happy to add them to the documentation, but I think > it's otherwise reasonable. > > However, I'm not sure if a completely new provider is the way to go. Can > you provide more details about what's insufficient about cqsub? > > Mihael > > On Fri, 2014-08-22 at 22:00 -0500, Michael Wilde wrote: > > Here's a pointer to the docs for this. I should re-read before I ask > > for more details :) > > > > https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1245 > > > > --- Comment #4 from Mihael Hategan 2014-06-10 > 01:05:00 --- > > Cog trunk r3918 has this. There is an example in > > provider-localscheduler/examples as well as documentation in > > provider-localscheduler/shell-provider.README > > --- > > > > - Mike > > > > > > On 8/22/14, 9:04 PM, Michael Wilde wrote: > > > We discovered today that the current cobalt provider needs to be > > > converted from using cqsub to the BG/Q's version of Cobalt qsub. > > > > > > This seems like an excellent opportunity to use the new templated > > > local scheduler provider. > > > > > > Mihael, can you provide instructions for how to do this? > > > > > > As I recall, the prior info you sent regarding this (when you first > > > developed this feature) needed further clarification. > > > > > > Thanks, > > > > > > - Mike > > > > > > > > > > > > On 8/22/14, 1:44 PM, Ketan Maheshwari wrote: > > >> PS. The current java is from IBM: > > >> > > >> $ java -version > > >> java version "1.6.0" > > >> Java(TM) SE Runtime Environment (build pxp6460sr15-20131017_01(SR15)) > > >> IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux ppc64-64 > > >> jvmxp6460sr15-20131016_170922 (JIT enabled, AOT enabled) > > >> J9VM - 20131016_170922 > > >> JIT - r9_20130920_46510ifx2 > > >> GC - GA24_Java6_SR15_20131016_1337_B170922) > > >> JCL - 20131015_01 > > >> > > >> > > >> On Fri, Aug 22, 2014 at 1:41 PM, Ketan Maheshwari > >> > wrote: > > >> > > >> Hi, > > >> > > >> I am getting this error trying to build Swift trunk on BG Vesta: > > >> > > >> compile: > > >> [echo] [util]: COMPILE > > >> [mkdir] Created dir: > > >> /gpfs/vesta-home/ketan/swift-devel/cog/modules/util/build > > >> [javac] Compiling 56 source files to > > >> /gpfs/vesta-home/ketan/swift-devel/cog/modules/util/build > > >> [javac] > > >> > /gpfs/vesta-home/ketan/swift-devel/cog/modules/util/src/org/globus/cog/util/concurrent/FileLock.java:91: > > >> cannot find symbol > > >> [javac] symbol : class VMManagement > > >> [javac] location: package sun.management > > >> [javac] sun.management.VMManagement mgmt = > > >> (sun.management.VMManagement) jvm.get(runtime); > > >> [javac] ^ > > >> [javac] > > >> > /gpfs/vesta-home/ketan/swift-devel/cog/modules/util/src/org/globus/cog/util/concurrent/FileLock.java:91: > > >> cannot find symbol > > >> [javac] symbol : class VMManagement > > >> [javac] location: package sun.management > > >> [javac] sun.management.VMManagement mgmt = > > >> (sun.management.VMManagement) jvm.get(runtime); > > >> [javac] ^ > > >> [javac] Note: Some input files use or override a deprecated > API. > > >> [javac] Note: Recompile with -Xlint:deprecation for details. > > >> [javac] Note: Some input files use unchecked or unsafe > > >> operations. > > >> [javac] Note: Recompile with -Xlint:unchecked for details. > > >> [javac] 2 errors > > >> > > >> Doe this means I need sun java on Vesta? Does anyone know if Sun > > >> java is available for PPC64 architecture? > > >> > > >> Thanks, > > >> Ketan > > >> > > >> > > >> > > >> > > >> _______________________________________________ > > >> Swift-devel mailing list > > >> Swift-devel at ci.uchicago.edu > > >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > > > > -- > > > Michael Wilde > > > Mathematics and Computer Science Computation Institute > > > Argonne National Laboratory The University of Chicago > > > > > > > > > _______________________________________________ > > > Swift-devel mailing list > > > Swift-devel at ci.uchicago.edu > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.g.armstrong at gmail.com Wed Aug 27 21:22:51 2014 From: tim.g.armstrong at gmail.com (Tim Armstrong) Date: Wed, 27 Aug 2014 21:22:51 -0500 Subject: [Swift-devel] coaster-service warning messages Message-ID: I've been experimenting with the coaster C client on midway and have been seeing a lot of warning messages from the coaster service. I wanted to know if this is a known problem, or get some guidance on how to try to resolve it. I'm running the latest github master version. I've been starting the coaster service with active coasters: export GLOBUS_HOSTNAME=172.25.180.72 coaster-service -nosec -p 65001 I then have the coaster C client connect to it and submit jobs, with the following settings: jobManager=slurm,jobQueue=sandyb,tasksperworker=16. The jobs have mostly been completing successfully, but I've also seen some instability and failures. I don't know if it's related to the many warnings in the service log (attached) , e.g. this one: 2014-08-28 01:06:34,174+0000 WARN TaskNotifier Client could not properly process notification: null java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113) at java.net.SocketOutputStream.write(SocketOutputStream.java:147) at org.globus.cog.coaster.channels.Sender.send(Sender.java:149) at org.globus.cog.coaster.channels.Sender.run(Sender.java:85) or this one: 2014-08-28 01:06:51,872+0000 WARN RemoteLogger Failed to send remote log message: BLOCK_SHUTDOWN id=0828-0601150-000000 org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:253) at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:274) at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:245) at org.globus.cog.abstraction.coaster.rlog.RemoteLogger.log(RemoteLogger.java:53) at org.globus.cog.abstraction.coaster.service.job.manager.Block.shutdown(Block.java:303) at org.globus.cog.abstraction.coaster.service.job.manager.Block.shutdownIfEmpty(Block.java:236) at org.globus.cog.abstraction.coaster.service.job.manager.Block.suspend(Block.java:576) at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.removeIdleBlocks(BlockQueueProcessor.java:472) at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.updatePlan(BlockQueueProcessor.java:750) at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.run(BlockQueueProcessor.java:161) Any guidance or thoughts would be appreciated. - Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: cps-2014-08-28_01-06-05.log Type: text/x-log Size: 161663 bytes Desc: not available URL: From ketan at mcs.anl.gov Fri Aug 29 13:36:13 2014 From: ketan at mcs.anl.gov (Ketan Maheshwari) Date: Fri, 29 Aug 2014 13:36:13 -0500 Subject: [Swift-devel] passing sites attribute to _swiftwrap Message-ID: Hi Mihael, All, I am trying to implement the subblock feature of Cobalt provider in Swift. While I am exploring and experimenting with the provider code, I thought to ask about your suggestions on this. The approach I am taking is to have an sites attribute just like PBS's ProviderAttributes for aprun. Next thing, this value needs to be passed to _swiftwap so that it can invoke the Cobalt runjob command after invoking the right subblock logic, which I am planning to move to _swiftwrap too. Any suggestions as to 1) if you think this is the right way to go about implementing subblock jobs and 2) What part of the code should I be looking into to make this work. Thanks, Ketan -------------- next part -------------- An HTML attachment was scrubbed... URL: