From hategan at mcs.anl.gov  Thu Aug  2 01:56:08 2012
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 01 Aug 2012 23:56:08 -0700
Subject: [Swift-devel] multiple proxy support
Message-ID: <1343890568.27574.4.camel@blabla>

There is code now in trunk to allow different GSI proxies to be used for
different hosts. 

It works as follows:
create ~/.globus/proxy.mapping and populate it with lines of the form
"hostname=/path/to/proxy"

GSI based operations (the kind from cog4, not purely jglobus) going to
"hostname" will try to load the proxy from "/path/to/proxy" instead of
the default "/tmp/x509_u<userid>".

(this is as of cog trunk r3444).

Mihael



From wilde at mcs.anl.gov  Thu Aug  2 04:44:53 2012
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 2 Aug 2012 04:44:53 -0500 (CDT)
Subject: [Swift-devel] multiple proxy support
In-Reply-To: <1343890568.27574.4.camel@blabla>
Message-ID: <528109874.46818.1343900693163.JavaMail.root@zimbra.anl.gov>

Beautiful - thanks, Mihael!

David, can you test this using a CI proxy on PADS and an LCRC proxy on Fusion (or ALCF proxy on Eureka)?

Kyle, can we also test against the CMTS GCMU server? If we can demonstrate that with globus-url-copy then it should now work from Swift.

Mihael, how do we handle the issue of CA certs? These all need to be in the X509_CERT_DIR/X509_CA_DIR of the swift client process, I assume?

- Mike


----- Original Message -----
> From: "Mihael Hategan" <hategan at mcs.anl.gov>
> To: "Swift Devel" <swift-devel at ci.uchicago.edu>
> Sent: Thursday, August 2, 2012 1:56:08 AM
> Subject: [Swift-devel] multiple proxy support
> There is code now in trunk to allow different GSI proxies to be used
> for
> different hosts.
> 
> It works as follows:
> create ~/.globus/proxy.mapping and populate it with lines of the form
> "hostname=/path/to/proxy"
> 
> GSI based operations (the kind from cog4, not purely jglobus) going to
> "hostname" will try to load the proxy from "/path/to/proxy" instead of
> the default "/tmp/x509_u<userid>".
> 
> (this is as of cog trunk r3444).
> 
> Mihael
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From iraicu at cs.iit.edu  Thu Aug  2 09:51:55 2012
From: iraicu at cs.iit.edu (Ioan Raicu)
Date: Thu, 02 Aug 2012 09:51:55 -0500
Subject: [Swift-devel] CFP: 5th Workshop on Many-Task Computing on Grids and
 Supercomputers (MTAGS) 2012 -- at IEEE/ACM Supercomputing 2012
Message-ID: <501A940B.70001@cs.iit.edu>

Call for Papers

---------------------------------------------------------------------------------------
The 5th Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS) 2012
http://datasys.cs.iit.edu/events/MTAGS12/
---------------------------------------------------------------------------------------
November 12th, 2012
Salt Lake City, Utah, USA

Co-located with with IEEE/ACM International Conference for
High Performance Computing, Networking, Storage and Analysis (SC12)

=======================================================================================
The 5th workshop on Many-Task Computing on Grids and Supercomputers (MTAGS) will
provide the scientific community a dedicated forum for presenting new research,
development, and deployment efforts of large-scale many-task computing (MTC)
applications on large scale clusters, Grids, Supercomputers, and Cloud Computing
infrastructure. MTC, the theme of the workshop encompasses loosely coupled applications,
which are generally composed of many tasks (both independent and dependent tasks) to
achieve some larger application goal.  This workshop will cover challenges that can
hamper efficiency and utilization in running applications on large-scale systems, such
as local resource manager scalability and granularity, efficient utilization of raw
hardware, parallel file system contention and scalability, data management, I/O
management, reliability at scale, and application scalability. We welcome paper
submissions on all theoretical, simulations, and systems topics related to MTC, but we
give special consideration to papers addressing petascale to exascale challenges.
Papers will be peer-reviewed, and accepted papers will be published in the workshop
proceedings as part of the IEEE digital library (pending approval). The workshop will
be co-located with the IEEE/ACM Supercomputing 2012 Conference in Salt Lake City Utah
on November 12th, 2012.
  
For more information, please see http://datasys.cs.iit.edu/events/MTAGS12/. For more
information on past workshops, please see MTAGS11, MTAGS10, MTAGS09, and MTAGS08. We also
ran a Special Issue on Many-Task Computing in the IEEE Transactions on Parallel and
Distributed Systems (TPDS) which has appeared in June 2011; the proceedings can be found
online at http://www.computer.org/portal/web/csdl/abs/trans/td/2011/06/ttd201106toc.htm.
We, the workshop organizers, also published a highly relevant paper that defines
Many-Task Computing which was published in MTAGS08, titled "Many-Task Computing for
Grids and Supercomputers"
(http://www.cs.iit.edu/~iraicu/research/publications/2008_MTAGS08_MTC.pdf); we encourage
potential authors to read this paper.

Topics
---------------------------------------------------------------------------------------
We invite the submission of original work that is related to the topics below. The
papers can be either short (4 pages) position papers, or long (8 pages) research
papers. Topics of interest include (in the context of Many-Task Computing):
* Compute Resource Management
   * Scheduling
   * Job execution frameworks
   * Local resource manager extensions
   * Performance evaluation of resource managers in use on large scale systems
   * Dynamic resource provisioning
   * Techniques to manage many-core resources and/or GPUs
   * Challenges and opportunities in running many-task workloads on HPC systems
   * Challenges and opportunities in running many-task workloads on Cloud Computing
     infrastructure
* Storage architectures and implementations
   * Distributed file systems
   * Parallel file systems
   * Distributed meta-data management
   * Content distribution systems for large data
   * Data caching frameworks and techniques
   * Data management within and across data centers
   * Data-aware scheduling
   * Data-intensive computing applications
   * Eventual-consistency storage usage and management
* Programming models and tools
   * Map-reduce and its generalizations
   * Many-task computing middleware and applications
   * Parallel programming frameworks
   * Ensemble MPI techniques and frameworks
   * Service-oriented science applications
* Large-Scale Workflow Systems
   * Workflow system performance and scalability analysis
   * Scalability of workflow systems
   * Workflow infrastructure and e-Science middleware
   * Programming Paradigms and Models
* Large-Scale Many-Task Applications
   * High-throughput computing (HTC) applications
   * Data-intensive applications
   * Quasi-supercomputing applications, deployments, and experiences
   * Performance Evaluation
* Performance evaluation
   * Real systems
   * Simulations
   * Reliability of large systems
* How MTC Addresses Challenges of Petascale and Exascale Computing
   * Concurrency & Programmability
   * I/O & Memory
   * Energy
   * Resilience
   * Heterogeneity

Paper Submission and Publication
---------------------------------------------------------------------------------------
Authors are invited to submit papers with unpublished, original work of not more than 8
pages of double column text using single spaced 10 point size on 8.5 x 11 inch pages,
as per IEEE 8.5 x 11 manuscript guidelines; document templates can be found at
http://www.ieee.org/conferences_events/conferences/publishing/templates.html. We are
also seeking position papers of no more than 4 pages in length. The final 4/8 page
papers in PDF format must be submitted online at
https://cmt.research.microsoft.com/MTAGS2012/ before the deadline. Papers will be
peer-reviewed, and accepted papers will be published in the workshop proceedings as part
of the IEEE digital library (pending approval). Notifications of the paper decisions
will be sent out by October 12th, 2012. Selected excellent work may be eligible for
additional post-conference publication as journal articles or book chapters, such as the
previous Special Issue on Many-Task Computing in the IEEE Transactions on Parallel and
Distributed Systems (TPDS) which has appeared in June 2011. Submission implies the
willingness of at least one of the authors to register and present the paper. For more
information, please http://datasys.cs.iit.edu/events/MTAGS12/, or send email to
mtags12-chairs at datasys.cs.iit.edu.

Important Dates
---------------------------------------------------------------------------------------
* Abstract submission: September 10th, 2012 (11:59PM PST)
* Paper submission: September 17th, 2012 (11:59PM PST)
* Acceptance notification: October 12th, 2012
* Final papers due: November 7th, 2012

Committee Members
---------------------------------------------------------------------------------------
Workshop Chairs (mtags12-chairs at datasys.cs.iit.edu)
* Ioan Raicu, Illinois Institute of Technology & Argonne National Laboratory
* Ian Foster, University of Chicago & Argonne National Laboratory
* Yong Zhao, University of Electronic Science and Technology of China

Steering Committee
* David Abramson, Monash University, Australia
* Jack Dongara, University of Tennessee, USA
* Geoffrey Fox, Indiana University, USA
* Manish Parashar, Rutgers University, USA
* Marc Snir, University of Illinois at Urbana Champaign, USA
* Xian-He Sun, Illinois Institute of Technology, USA
* Weimin Zheng, Tsinghua University, China

Publicity Chair (mtags12-publicity at datasys.cs.iit.edu)
* Zhao Zhang, University of Chicago, USA

Program Committee Chair (mtags12-pc-chair at datasys.cs.iit.edu)
* Justin Wozniak, Argonne National Laboratory, USA

Technical Committee
* Roger Barga, Microsoft Research, USA
* Kyle Chard, University of Chicago, USA
* Dennis Gannon, Microsoft Research, USA
* Florin Isaila, Universidad Carlos III de Madrid, Spain
* Hui Jin, Oracle Corporation, USA
* Daniel S. Katz, University of Chicago, USA
* Zhiling Lan, Illinois Institute of Technology, USA
* Mike Lang, Los Alamos National Laboratory, USA
* Gregor von Laszewski, Indiana University, USA
* Reagan Moore, University of North Carolina, Chappel Hill, USA
* David O'Hallaron, Carnegie Mellon University, Intel Labs, USA
* Marlon Pierce, Indiana University, USA
* Judy Qiu, Indiana University, USA
* Lavanya Ramakrishnan, Lawrence Berkeley National Laboratory, USA
* Kui Ren, SUNY Buffalo, USA
* Matei Ripeanu, University of British Columbia, Canada
* Wei Tang, Argonne National Laboratory, USA
* Valerie Taylor, Texas A&M, USA
* Ken Yocum, University of California, San Diego, USA
* Zhifeng Yun, Louisiana State University, USA
* Zhao Zhang, University of Chicago, USA

-- 
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor, Illinois Institute of Technology (IIT)
Guest Research Faculty, Argonne National Laboratory (ANL)
=================================================================
Data-Intensive Distributed Systems Laboratory, CS/IIT
Distributed Systems Laboratory, MCS/ANL
=================================================================
Cel:    1-847-722-0876
Office: 1-312-567-5704
Email:  iraicu at cs.iit.edu
Web:    http://www.cs.iit.edu/~iraicu/
Web:    http://datasys.cs.iit.edu/
=================================================================
=================================================================



From davidk at ci.uchicago.edu  Thu Aug  2 12:24:07 2012
From: davidk at ci.uchicago.edu (David Kelly)
Date: Thu, 2 Aug 2012 12:24:07 -0500 (CDT)
Subject: [Swift-devel] Changing ppn on a per-application basis
In-Reply-To: <588286214.152302.1343926998743.JavaMail.root@zimbra-mb2.anl.gov>
Message-ID: <1250837372.152423.1343928247149.JavaMail.root@zimbra-mb2.anl.gov>

Hello,

I have a user who would like to change ppn on a per-application basis. Is this possible right now? As far as I know, the only way to do this is currently is to create separate pool entries with varying ppn values.

The example he gave me is this: the cluster he is working on supports node sharing. For some applications he may want to use all available processors. For other applications, especially those which use GPUs to do the heavy lifting, he may only want 1 ppn and leave the other processors for other users. I can also envision cases where ppn may need to be adjusted based on memory requirements or possibly based on some data that isn't known until the program runs.

Mike and I were talking about this the other day. One of the ideas he suggested was adding a new value like stdout to the app definition. Something like this I'd imagine:

app (file o) cat (file i, int myppn)
{
  cat @i stdout=@o ppn=myppn;
}

>From what I can gather, the user is only interested in ppn at the moment. But it may be useful to eventually have a generic way to override all settings in sites.xml from an app function.

David 


From wilde at mcs.anl.gov  Thu Aug  2 12:39:47 2012
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 2 Aug 2012 12:39:47 -0500 (CDT)
Subject: [Swift-devel] Changing ppn on a per-application basis
In-Reply-To: <1250837372.152423.1343928247149.JavaMail.root@zimbra-mb2.anl.gov>
Message-ID: <1553283542.48019.1343929187189.JavaMail.root@zimbra.anl.gov>

David, when we discussed this I couldnt recall the recent feature that Justin added for this. I found it when I saw this post. Its at:

http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html#_dynamic_profiles

Seems to do what you need; I suspect it needs testing.

- Mike


----- Original Message -----
> From: "David Kelly" <davidk at ci.uchicago.edu>
> To: swift-devel at ci.uchicago.edu
> Sent: Thursday, August 2, 2012 12:24:07 PM
> Subject: [Swift-devel] Changing ppn on a per-application basis
> Hello,
> 
> I have a user who would like to change ppn on a per-application basis.
> Is this possible right now? As far as I know, the only way to do this
> is currently is to create separate pool entries with varying ppn
> values.
> 
> The example he gave me is this: the cluster he is working on supports
> node sharing. For some applications he may want to use all available
> processors. For other applications, especially those which use GPUs to
> do the heavy lifting, he may only want 1 ppn and leave the other
> processors for other users. I can also envision cases where ppn may
> need to be adjusted based on memory requirements or possibly based on
> some data that isn't known until the program runs.
> 
> Mike and I were talking about this the other day. One of the ideas he
> suggested was adding a new value like stdout to the app definition.
> Something like this I'd imagine:
> 
> app (file o) cat (file i, int myppn)
> {
> cat @i stdout=@o ppn=myppn;
> }
> 
> From what I can gather, the user is only interested in ppn at the
> moment. But it may be useful to eventually have a generic way to
> override all settings in sites.xml from an app function.
> 
> David
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From davidk at ci.uchicago.edu  Thu Aug  2 14:04:37 2012
From: davidk at ci.uchicago.edu (David Kelly)
Date: Thu, 2 Aug 2012 14:04:37 -0500 (CDT)
Subject: [Swift-devel] Changing ppn on a per-application basis
In-Reply-To: <1553283542.48019.1343929187189.JavaMail.root@zimbra.anl.gov>
Message-ID: <1805660857.152961.1343934277573.JavaMail.root@zimbra-mb2.anl.gov>

Mike,

Thank you. The combination of setting ppn and jobsPerNode in the app function seems to work and do exactly what I need.

David

----- Original Message -----
> From: "Michael Wilde" <wilde at mcs.anl.gov>
> To: "David Kelly" <davidk at ci.uchicago.edu>
> Cc: swift-devel at ci.uchicago.edu
> Sent: Thursday, August 2, 2012 12:39:47 PM
> Subject: Re: [Swift-devel] Changing ppn on a per-application basis
> David, when we discussed this I couldnt recall the recent feature that
> Justin added for this. I found it when I saw this post. Its at:
> 
> http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html#_dynamic_profiles
> 
> Seems to do what you need; I suspect it needs testing.
> 
> - Mike
> 
> 
> ----- Original Message -----
> > From: "David Kelly" <davidk at ci.uchicago.edu>
> > To: swift-devel at ci.uchicago.edu
> > Sent: Thursday, August 2, 2012 12:24:07 PM
> > Subject: [Swift-devel] Changing ppn on a per-application basis
> > Hello,
> >
> > I have a user who would like to change ppn on a per-application
> > basis.
> > Is this possible right now? As far as I know, the only way to do
> > this
> > is currently is to create separate pool entries with varying ppn
> > values.
> >
> > The example he gave me is this: the cluster he is working on
> > supports
> > node sharing. For some applications he may want to use all available
> > processors. For other applications, especially those which use GPUs
> > to
> > do the heavy lifting, he may only want 1 ppn and leave the other
> > processors for other users. I can also envision cases where ppn may
> > need to be adjusted based on memory requirements or possibly based
> > on
> > some data that isn't known until the program runs.
> >
> > Mike and I were talking about this the other day. One of the ideas
> > he
> > suggested was adding a new value like stdout to the app definition.
> > Something like this I'd imagine:
> >
> > app (file o) cat (file i, int myppn)
> > {
> > cat @i stdout=@o ppn=myppn;
> > }
> >
> > From what I can gather, the user is only interested in ppn at the
> > moment. But it may be useful to eventually have a generic way to
> > override all settings in sites.xml from an app function.
> >
> > David
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory


From wilde at mcs.anl.gov  Fri Aug  3 23:19:49 2012
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Fri, 3 Aug 2012 23:19:49 -0500 (CDT)
Subject: [Swift-devel] Fwd:  hang checker updates
In-Reply-To: <1344043817.14696.28.camel@blabla>
Message-ID: <1608248859.51033.1344053989016.JavaMail.root@zimbra.anl.gov>

Mihael, good points.  Much of this belongs in the user guide. I'll paste it into a ticket.

- Mike

----- Forwarded Message -----
From: "Mihael Hategan" <hategan at mcs.anl.gov>
To: "Jared M Chase" <jared.chase at pnnl.gov>
Cc: "Karen L Schuchardt" <Karen.Schuchardt at pnnl.gov>, "Khushbu Agarwal" <Khushbu.Agarwal at pnnl.gov>, "Michael Wilde" <wilde at mcs.anl.gov>, "David Kelly" <davidk at ci.uchicago.edu>, "Justin M. Wozniak" <wozniak at mcs.anl.gov>
Sent: Friday, August 3, 2012 8:30:17 PM
Subject: RE: [Swift-devel] hang checker updates

On Fri, 2012-08-03 at 16:36 -0700, Chase, Jared M wrote:
> The 6 hour run ran for around 4 hours until we got an exception.  The
> exception says it is looking for a file under the directory that we
> are submitting the job (/scratch/scratchdirs/jchase/hybrid) as opposed
> to the work directory
> (/scratch/scratchdirs/jchase/hybrid/work/hw-clean-jared-20120803-1135-uq7t7sae).  Also, there is another file not found exception for one of the info directories ...


I'm having some trouble following this. As far as I can tell, it should
not have gotten as far as it did. It looks like every iteration requires
a "run-n/sph-1/sph.output.h5part", but it looks like iteration 80 is the
last one where such a file is produced.

mike at blabla:~/tmp/swift-bugs/i$ cat
hw-clean-jared-20120803-1135-uq7t7sae.log |egrep
'FILE_STAGE_OUT_START.*h5part.*sph-1 '|awk '{print $9}'

So I suspect that you get this far because you have those files from
previous runs.

I'll make a few observations that may not be immediately apparent from
the swift documentation:

1. with the exception of iterate{} (whose current implementation happens
to order iterations sequentially), execution is based purely on data
dependencies. So in the following example, all instances of trace("A")
will run long before the apps run because they have no dependencies they
need to wait for:
   foreach i in [1:4] {
     outf[i] = cat(inf);
     trace(outf[i]); // you will see these as the cat apps complete
     trace("A"); //you will see 4 "trace: A" in the beginning
   }
2. In the same spirit, trace("A");trace("B"); will either result in "A"
being printed first or in "B" being printed first. There is nothing to
enforce a particular sequence, since there is no dependency between the
two. Syntactic execution ordering does not apply.
3. You are using implicit dependencies in that in an iteration, you map
files that you expect to be there from the previous iteration. The
"proper" way to do it is to explicitly pass those files (not just names)
as arguments between iterations, although I understand that you may have
done the former to go around a swift bug (which is fixed in the latest
swift package I sent). I wouldn't change this now if you are comfortable
with the current scheme, but you need to make sure that things
implicitly passed between iterations are actually created.

> 
> Caused by: org.globus.cog.abstraction.impl.file.FileNotFoundException:
> File not
> found: /scratch/scratchdirs/jchase/hybrid/work/hw-clean-jared-20120803-1135-uq7t7sae/info/j/pg-jp0go4vk-info

Whenever a job fails, swift tries to gather logs related to that job for
post-mortem troubleshooting. Sometimes those logs are not available, and
errors in the attempts to transfer them are logged, but they do not
represent a primary type of error.

Mihael


-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From iraicu at cs.iit.edu  Tue Aug  7 22:47:07 2012
From: iraicu at cs.iit.edu (Ioan Raicu)
Date: Tue, 07 Aug 2012 22:47:07 -0500
Subject: [Swift-devel] Call for Participation: The 9th ACM International
 Conference on Autonomic Computing (ICAC 2012) -- September 17-21,
 San Jose CA
Message-ID: <5021E13B.108@cs.iit.edu>

**********************************************************************

CALL FOR PARTICIPATION
======================

The 9th ACM International Conference on Autonomic Computing (ICAC 2012)

San Jose, California, USA
September 17-21, 2012
http://icac2012.cs.fiu.edu

Sponsored by ACM

**********************************************************************

Online registration is open at
http://icac2012.cs.fiu.edu/registration.shtm
Reduced fees are available for those registering by September 2, 2012.

This year's technical program features:

- 3 distinguished keynote speakers:
* Dr. Amin Vahdat, Google/UCSD
* Dr. Subutai Ahmad (VP Engineering), Numenta
* Dr. Eitan Frachtenberg, Facebook

- 24 outstanding technical papers (15 full + 9 short):
* covering core and emerging topics such as clouds, virtualization,
control, monitoring and diagnosis, and energy
* Half of the papers involve authors from industry or government labs

- 4 co-located workshops covering hot topics in:
* Feedback Computing
* Self-Aware Internet of Things
* Management of Big Data Systems
* Federated Clouds



The Conference will be held at the Fairmont Hotel in downtown San Jose, 
CA, USA.


**********************************************************************

IMPORTANT DATES
===============

Early registration deadline: September 2, 2012
Hotel special rate deadline: September 7, 2012

**********************************************************************

CORPORATE SPONSORS
==================

Gold Level Partner: IBM
Conference partners: VMware, HP, Neustar
PhD Student Sponsor: Google
Other Sponsor: NEC Labs

**********************************************************************

PRELIMINARY PROGRAM
===================

======================================================================

MONDAY, SEPTEMBER 17, 2012 - WORKSHOPS

Feedback Computing 2012
Self-Aware Internet of Things 2012

======================================================================

TUESDAY, SEPTEMBER 18, 2012 - MAIN CONFERENCE

8:00AM ? 8:45AM
Registration

8:45AM ? 9:00AM
Welcome Remarks

9:00AM ? 10:00AM
Keynote Talk I

Symbiosis in Scale Out Networking and Data Management
Amin Vahdat, Google and University of California, San Diego

10:00AM ? 10.30AM
Break

10:30AM ? 12:00PM
Session : Virtualization

Net-Cohort: Detecting and Managing VM Ensembles in Virtualized Data Centers
Liting Hu, Karsten Schwan (Georgia Institute of Technology);
Ajay Gulati (VMware); Junjie Zhang, Chengwei Wang (Georgia Institute
of Technology)

Application-aware Cross-layer Virtual Machine Resource Management
Lixi Wang, Jing Xu, Ming Zhao (Florida International University)

Shifting GEARS to Enable Guest-context Virtual Services
Kyle Hale, Lei Xia, Peter Dinda (Northwestern University)

12:00PM-1:30PM
Lunch

----------------------------------------------------------------------
1:30PM - 3:30PM
Session: Performance and Resource Management

When Average is Not Average: Large Response Time Fluctuations in N-tier 
Systems
Qingyang Wang (Georgia Institute of Technology); Yasuhiko Kanemasa,
Motoyuki Kawaba (Fujitsu Laboratories Ltd.); Calton Pu (Georgia Institute
of Technology)

Provisioning Multi-tier Cloud Applications Using Statistical Bounds on
Sojourn Time
Upendra Sharma, Prashant Shenoy, Don Towsley (University of 
Massachusetts Amherst)

Automated Profiling and Resource Management of Pig Programs for Meeting
Service Level Objectives
Zhuoyao Zhang (University of Pennsylvania); Ludmila Cherkasova 
(Hewlett-Packard Labs);
Abhishek Verma (University of Illinois at Urbana-Champaign);
Boon Thau Loo (University of Pennsylvania)

AROMA: Automated Resource Allocation and Configuration of MapReduce 
Environment
in the Cloud
Palden Lama, Xiaobo Zhou (University of Colorado at Colorado Springs)

3:30PM-4:00PM
Break

4:00PM ? 5:15PM
Short Papers I

Locomotion at Location: When the Rubber hits the Road
Gerold Hoelzl, Marc Kurz, Alois Ferscha (Johannes Kepler University 
Linz, Austria)

An Autonomic Resource Provisioning Framework for Mobile Computing Grids
Hariharasudhan Viswanathan, Eun Kyung Lee, Ivan Rodero, Dario Pompili 
(Rutgers University)

A Self-Tuning Self-Optimizing Approach for Automated Network Anomaly 
Detection Systems
Dennis Ippoliti, Xiaobo Zhou (University of Colorado at Colorado Springs)

Offline and On-Demand Event Correlation for Operations Management of 
Large Scale IT Systems
Chetan Gupta (Hewlett-Packard Labs)

PowerTracer: Tracing Requests in Multi-tier Services to Diagnose Energy 
Inefficiency
Gang Lu, Jianfeng Zhan (Institute of Computing Technology, Chinese 
Academy of Sciences);
Haining Wang (College of William and Mary); Lin Yuan (Institute of 
Computing Technology,
Chinese Academy of Sciences); Chuliang Weng (Shanghai Jiao Tong 
University, China)

6:00PM - 9:00PM
Conference Dinner

======================================================================

WEDNESDAY, SEPTEMBER 19 - MAIN CONFERENCE

8:00AM ? 9:00AM
Registration

9:00AM ? 10:00AM
Keynote Talk II

Automated Machine Learning For Autonomic Computing
Subutai Ahmad, VP Engineering, Numenta

10:00AM - 10:30AM
Break

10:30AM ? 12:00PM
Session: Control-Based Approaches

Budget-based Control for Interactive Services with Adaptive Execution
Yuxiong He, Zihao Ye, Qiang Fu, Sameh Elnikety (Microsoft Research)

On the Design of Decentralized Control Architectures for Workload 
Consolidation in
Large-Scale Server Clusters
Rui Wang, Nagarajan Kandasamy (Drexel University)

Transactional Auto Scaler: Elastic Scaling of In-Memory Transactional 
Data Grids
Diego Didona, Paolo Romano (Instituto Superior T?cnico/INESC-ID); 
Sebastiano Peluso,
Francesco Quaglia (Sapienza, Universit? di Roma)

12:00PM - 1:30PM
Lunch

----------------------------------------------------------------------

1:30PM - 2:30PM
Session: Energy

Adaptive Green Hosting
Nan Deng, Christopher Stewart, Jaimie Kelley (The Ohio State 
University); Daniel Gmach,
Martin Arlitt (Hewlett Packard Labs)

Dynamic Energy-Aware Capacity Provisioning for Cloud Computing Environments
Qi Zhang, Mohamed Faten Zhani (University of Waterloo); Shuo Zhang 
(National University
of Defense Technology); Quanyan Zhu (University of Illinois at 
Urbana-Champaign);
Raouf Boutaba (University of Waterloo); Joseph L. Hellerstein (Google, 
Inc.)

2:30PM - 3:30PM
Short Papers II

VESPA: Multi-Layered Self-Protection for Cloud Resources
Aur?lien Wailly, Marc Lacoste (Orange Labs); Herv? Debar (T?l?com SudParis)

Usage Patterns in Multi-tenant Data Centers: a Temporal Perspective
Robert Birke, Lydia Y. Chen (IBM Research Zurich Lab); Evgenia Smirni 
(College of
William and Mary)

Toward Fast Eventual Consistency with Performance Guarantees
Feng Yan (College of William and Mary); Alma Riska (EMC Corporation); 
Evgenia Smirni (College
of William and Mary)

Optimal Autoscaling in the IaaS Cloud
Hamoun Ghanbari, Bradley Simmons, Marin Litoiu, Cornel Barna (York 
University);
Gabriel Iszlai (IBM Toronto)

3:30PM ? 4:00PM
Break

4:00PM - 6:00PM
Poster and Demo Session

6:00PM-9:00PM
Conference Outing (tentative)

======================================================================

THURSDAY, SEPTEMBER 20, 2012 - MAIN CONFERENCE

8:00AM ? 9:00AM
Registration

9:00AM ? 10:00AM
Keynote Talk III

High Efficiency at Web Scale
Eitan Frachtenberg, Facebook

10:00AM-10:30AM
Break

10:30AM - 12:00PM
Session: Diagnosis and Monitoring
Chair: TBD

3-Dimensional Root Cause Diagnosis via Co-analysi
Ziming Zheng, Li Yu, Zhiling Lan (Illinois Institute of Technology); 
Terry Jones
(Oak Ridge National Laboratory)

UBL: Unsupervised Behavior Learning for Predicting Performance Anomalies 
in Virtualized Cloud Systems
Daniel J. Dean, Hiep Nguyen, Xiaohui Gu (North Carolina State University)

Evaluating Compressive Sampling Strategies for Performance Monitoring of 
Data Centers
Tingshan Huang, Nagarajan Kandasamy, Harish Sethu (Drexel University)

12:00PM
Adjourn

======================================================================

FRIDAY, SEPTEMBER 21, 2012 - WORKSHOPS

Management of Big Data Systems 2012
Federated Clouds 2012

**********************************************************************

ORGANIZERS
==========

GENERAL CHAIR
Dejan Milojicic, HP Labs

PROGRAM CHAIRS
Dongyan Xu, Purdue University
Vanish Talwar, HP Labs

INDUSTRY CHAIR
Xiaoyun Zhu, VMware

WORKSHOPS CHAIR
Fred Douglis, EMC

POSTERS/DEMO/EXHIBITS CHAIR
Eno Thereska, Microsoft Research

FINANCE CHAIR
Michael Kozuch, Intel

LOCAL ARRANGEMENT CHAIR
Jessica Blaine

PUBLICITY CHAIRS
Daniel Batista, University of S?o Paulo
Vartan Padaryan, ISP/Russian Academy of Sci.
Ioan Raicu, Illinois Inst. of Technology
Jianfeng Zhan, ICT/Chinese Academy of Sci.
Ming Zhao, Florida Intl. University

PROGRAM COMMITTEE
Tarek Abdelzaher, UIUC
Umesh Bellur, IIT, Bombay
Ken Birman, Cornell University
Rajkumar Buyya, Univ. of Melbourne
Rocky Chang, Hong Kong Polytechnic University
Yuan Chen, HP Labs
Alva Couch, Tufts University
Peter Dinda, Northwestern University
Fred Douglis, EMC
Renato Figueiredo, University of Florida
Mohamed Hefeeda, QCRI
Joe Hellerstein, Google
Geoff Jiang, NEC Labs
Jeff Kephart, IBM Research
Emre Kiciman, Microsoft Research
Fabio Kon, University of S?o Paulo
Mike Kozuch, Intel Labs
Dejan Milojicic, HP Labs
Klara Nahrstedt, UIUC
Priya Narasimhan, CMU
Manish Parashar, Rutgers University
Ioan Raicu, Illinois Inst. of Technology
Omer Rana, Cardiff University
Masoud Sadjadi, Florida Intl. University
Richard Schlichting, AT&T Labs
Hartmut Schmeck, KIT
Karsten Schwan, Georgia Tech
Onn Shehory, IBM Research
Eno Thereska, Microsoft Research
Xiaoyun Zhu, VMware
**********************************************************************

-- 
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor, Illinois Institute of Technology (IIT)
Guest Research Faculty, Argonne National Laboratory (ANL)
=================================================================
Data-Intensive Distributed Systems Laboratory, CS/IIT
Distributed Systems Laboratory, MCS/ANL
=================================================================
Cel:    1-847-722-0876
Office: 1-312-567-5704
Email:  iraicu at cs.iit.edu
Web:    http://www.cs.iit.edu/~iraicu/
Web:    http://datasys.cs.iit.edu/
=================================================================
=================================================================



From wilde at mcs.anl.gov  Mon Aug 20 09:28:49 2012
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Mon, 20 Aug 2012 09:28:49 -0500 (CDT)
Subject: [Swift-devel] Fwd: [wg-all] New document published: Firewall
	Traversal Protocol
In-Reply-To: <20120820133032.GA18957@alaska.edu>
Message-ID: <1671992055.925.1345472929427.JavaMail.root@zimbra.anl.gov>

Perhaps relevant to Swift...

----- Forwarded Message -----
From: "Greg Newby" <gbnewby at alaska.edu>
To: "All WG" <wg-all at ogf.org>
Sent: Monday, August 20, 2012 8:30:32 AM
Subject: [wg-all] New document published: Firewall Traversal Protocol

OGF Community:

A new document has been published in the OGF series.  All OGF
documents (including any that are open for public comment) may be
found here:

  http://www.ogf.org/gf/docs/

* GFD-R-P.196 "Firewall Traversal Protocol (FiTP)," R. Niederberger via the Inffrastructure FVGA-WG.

 Abstract:
Firewalls control traffic flows between internal and external communication partners. Mostly traffic from inside to outside is allowed, but traffic coming from outside must be explicitly configured. The rules which packets may traverse the firewall and which not are normally configured manually by firewall administrators. To speed up such kind of access list changes, it would be desirable to dynamically signal access requests and automatically change those access lists. Though some protocols are inspectable by firewalls already like FTP, SIP and H.323, a general protocol, which could be used for signaling dynamically required access rules, is not available until now.

This paper proposes a standard protocol, which would allow such signaling in a secure manner. Firewalls which have installed a corresponding inspection module could be configured automatically, which would ease the configuration of such systems a lot.

The proposed protocol (FiTP) can be used in two ways. First of all, a firewall aware of FiTP, could automatically allow connections signaled by authorized users. Secondly, an intermediate solution could be implemented, so that firewalls unaware of FiTP could be configured by the server process, which is the end point of the FiTP control connection. Via this approach a smooth transition would be possible. Installations having old firewall hard- and/or software could use the new protocol already, before installing a system which is FiTP enabled.

  -- Greg Newby, OGF Editor

Dr. Gregory Newby, Director of the Arctic Region Supercomputing Center
Univ of Alaska Fairbanks-909 Koyukuk Dr-PO Box 756020-Fairbanks-AK 99775-6020
e: gbnewby at alaska.edu v: 907-450-8663 f: 907-450-8603 w: people.arsc.edu/~newby

--
  The wg-all mailing list is an umbrella list for all
  OGF mailing lists.  If you are subscribed to any OGF
  mailing list, you will receive mails to wg-all.

  You can unsubscribe from this list on

  http://www.ogf.org/cgi-bin/perl/unsubscribe-wg-all.pl

  wg-all mailing list
  wg-all at ogf.org
  https://www.ogf.org/mailman/listinfo/wg-all

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From iraicu at cs.iit.edu  Sun Aug 26 09:17:26 2012
From: iraicu at cs.iit.edu (Ioan Raicu)
Date: Sun, 26 Aug 2012 09:17:26 -0500
Subject: [Swift-devel] CFP: The 5th Workshop on Many-Task Computing on Grids
 and Supercomputers (MTAGS) 2012 -- co-located with SC12
Message-ID: <503A2FF6.3080203@cs.iit.edu>

Call for Papers

---------------------------------------------------------------------------------------
The 5th Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS) 2012
http://datasys.cs.iit.edu/events/MTAGS12/
---------------------------------------------------------------------------------------
November 12th, 2012
Salt Lake City, Utah, USA

Co-located with with IEEE/ACM International Conference for
High Performance Computing, Networking, Storage and Analysis (SC12)

=======================================================================================
The 5th workshop on Many-Task Computing on Grids and Supercomputers (MTAGS) will
provide the scientific community a dedicated forum for presenting new research,
development, and deployment efforts of large-scale many-task computing (MTC)
applications on large scale clusters, Grids, Supercomputers, and Cloud Computing
infrastructure. MTC, the theme of the workshop encompasses loosely coupled applications,
which are generally composed of many tasks (both independent and dependent tasks) to
achieve some larger application goal.  This workshop will cover challenges that can
hamper efficiency and utilization in running applications on large-scale systems, such
as local resource manager scalability and granularity, efficient utilization of raw
hardware, parallel file system contention and scalability, data management, I/O
management, reliability at scale, and application scalability. We welcome paper
submissions on all theoretical, simulations, and systems topics related to MTC, but we
give special consideration to papers addressing petascale to exascale challenges.
Papers will be peer-reviewed, and accepted papers will be published in the workshop
proceedings as part of the IEEE digital library (pending approval). The workshop will
be co-located with the IEEE/ACM Supercomputing 2012 Conference in Salt Lake City Utah
on November 12th, 2012.
  
For more information, please see http://datasys.cs.iit.edu/events/MTAGS12/. For more
information on past workshops, please see MTAGS11, MTAGS10, MTAGS09, and MTAGS08. We also
ran a Special Issue on Many-Task Computing in the IEEE Transactions on Parallel and
Distributed Systems (TPDS) which has appeared in June 2011; the proceedings can be found
online at http://www.computer.org/portal/web/csdl/abs/trans/td/2011/06/ttd201106toc.htm.
We, the workshop organizers, also published a highly relevant paper that defines
Many-Task Computing which was published in MTAGS08, titled "Many-Task Computing for
Grids and Supercomputers"
(http://www.cs.iit.edu/~iraicu/research/publications/2008_MTAGS08_MTC.pdf); we encourage
potential authors to read this paper.

Topics
---------------------------------------------------------------------------------------
We invite the submission of original work that is related to the topics below. The
papers can be either short (4 pages) position papers, or long (8 pages) research
papers. Topics of interest include (in the context of Many-Task Computing):
* Compute Resource Management
   * Scheduling
   * Job execution frameworks
   * Local resource manager extensions
   * Performance evaluation of resource managers in use on large scale systems
   * Dynamic resource provisioning
   * Techniques to manage many-core resources and/or GPUs
   * Challenges and opportunities in running many-task workloads on HPC systems
   * Challenges and opportunities in running many-task workloads on Cloud Computing
     infrastructure
* Storage architectures and implementations
   * Distributed file systems
   * Parallel file systems
   * Distributed meta-data management
   * Content distribution systems for large data
   * Data caching frameworks and techniques
   * Data management within and across data centers
   * Data-aware scheduling
   * Data-intensive computing applications
   * Eventual-consistency storage usage and management
* Programming models and tools
   * Map-reduce and its generalizations
   * Many-task computing middleware and applications
   * Parallel programming frameworks
   * Ensemble MPI techniques and frameworks
   * Service-oriented science applications
* Large-Scale Workflow Systems
   * Workflow system performance and scalability analysis
   * Scalability of workflow systems
   * Workflow infrastructure and e-Science middleware
   * Programming Paradigms and Models
* Large-Scale Many-Task Applications
   * High-throughput computing (HTC) applications
   * Data-intensive applications
   * Quasi-supercomputing applications, deployments, and experiences
   * Performance Evaluation
* Performance evaluation
   * Real systems
   * Simulations
   * Reliability of large systems
* How MTC Addresses Challenges of Petascale and Exascale Computing
   * Concurrency & Programmability
   * I/O & Memory
   * Energy
   * Resilience
   * Heterogeneity

Paper Submission and Publication
---------------------------------------------------------------------------------------
Authors are invited to submit papers with unpublished, original work of not more than 8
pages of double column text using single spaced 10 point size on 8.5 x 11 inch pages,
as per IEEE 8.5 x 11 manuscript guidelines; document templates can be found at
http://www.ieee.org/conferences_events/conferences/publishing/templates.html. We are
also seeking position papers of no more than 4 pages in length. The final 4/8 page
papers in PDF format must be submitted online at
https://cmt.research.microsoft.com/MTAGS2012/ before the deadline. Papers will be
peer-reviewed, and accepted papers will be published in the workshop proceedings as part
of the IEEE digital library (pending approval). Notifications of the paper decisions
will be sent out by October 12th, 2012. Selected excellent work may be eligible for
additional post-conference publication as journal articles or book chapters, such as the
previous Special Issue on Many-Task Computing in the IEEE Transactions on Parallel and
Distributed Systems (TPDS) which has appeared in June 2011. Submission implies the
willingness of at least one of the authors to register and present the paper. For more
information, please http://datasys.cs.iit.edu/events/MTAGS12/, or send email to
mtags12-chairs at datasys.cs.iit.edu.

Important Dates
---------------------------------------------------------------------------------------
* Abstract submission: September 10th, 2012 (11:59PM PST)
* Paper submission: September 17th, 2012 (11:59PM PST)
* Acceptance notification: October 12th, 2012
* Final papers due: November 7th, 2012

Committee Members
---------------------------------------------------------------------------------------
Workshop Chairs (mtags12-chairs at datasys.cs.iit.edu)
* Ioan Raicu, Illinois Institute of Technology & Argonne National Laboratory
* Ian Foster, University of Chicago & Argonne National Laboratory
* Yong Zhao, University of Electronic Science and Technology of China

Steering Committee
* David Abramson, Monash University, Australia
* Jack Dongara, University of Tennessee, USA
* Geoffrey Fox, Indiana University, USA
* Manish Parashar, Rutgers University, USA
* Marc Snir, Argonne National Laboratory & University of Illinois at Urbana Champaign, USA
* Xian-He Sun, Illinois Institute of Technology, USA
* Weimin Zheng, Tsinghua University, China

Publicity Chair (mtags12-publicity at datasys.cs.iit.edu)
* Zhao Zhang, University of Chicago, USA

Program Committee Chair (mtags12-pc-chair at datasys.cs.iit.edu)
* Justin Wozniak, Argonne National Laboratory, USA

Technical Committee
* Roger Barga, Microsoft Research, USA
* Mihai Budiu, Microsoft Research, USA
* Kyle Chard, University of Chicago, USA
* Yong Chen, Texas Tech University, USA
* Evangelinos Constantinos, Massachusetts Institute of Technology, USA
* John Dennis, National Center for Atmospheric Research, USA
* Catalin Dumitrescu, Fermi National Labs, USA
* Dennis Gannon, Microsoft Research, USA
* Indranil Gupta, University of Illinois at Urbana Champaign, USA
* Florin Isaila, Universidad Carlos III de Madrid, Spain
* Kamil Iskra, Argonne National Laboratory, USA
* Alexandru Iosup, Delft University of Technology, Netherlands
* Hui Jin, Oracle Corporation, USA
* Daniel S. Katz, University of Chicago & Argonne National Laboratory, USA
* Carl Kesselman, University of Southern California, USA
* Zhiling Lan, Illinois Institute of Technology, USA
* Mike Lang, Los Alamos National Laboratory, USA
* Gregor von Laszewski, Indiana University, USA
* Reagan Moore, University of North Carolina, Chappel Hill, USA
* Jose Moreira, IBM Research, USA
* Chris Moretti, Princeton University, USA
* David O'Hallaron, Carnegie Mellon University, Intel Labs, USA
* Marlon Pierce, Indiana University, USA
* Judy Qiu, Indiana University, USA
* Lavanya Ramakrishnan, Lawrence Berkeley National Laboratory, USA
* Kui Ren, SUNY Buffalo, USA
* Matei Ripeanu, University of British Columbia, Canada
* Karen Schuchardt, Pacific Northwest National Laboratory, USA
* Wei Tang, Argonne National Laboratory, USA
* Valerie Taylor, Texas A&M, USA
* Douglas Thain University of Notre Dame, USA
* Edward Walker, Whitworth University, USA
* Matthew Woitaszek, Occipital, Inc., USA
* Ken Yocum, University of California, San Diego, USA
* Zhifeng Yun, Louisiana State University, USA
* Zhao Zhang, University of Chicago, USA
* Ziming Zheng, Illinois Institute of Technology, USA

-- 
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor, Illinois Institute of Technology (IIT)
Guest Research Faculty, Argonne National Laboratory (ANL)
=================================================================
Data-Intensive Distributed Systems Laboratory, CS/IIT
Distributed Systems Laboratory, MCS/ANL
=================================================================
Cel:    1-847-722-0876
Office: 1-312-567-5704
Email:  iraicu at cs.iit.edu
Web:    http://www.cs.iit.edu/~iraicu/
Web:    http://datasys.cs.iit.edu/
=================================================================
=================================================================



From hategan at mcs.anl.gov  Mon Aug 27 01:13:52 2012
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 26 Aug 2012 23:13:52 -0700
Subject: [Swift-devel] coaster documentation
Message-ID: <1346048032.12314.4.camel@blabla>

I should have done this earlier, but here's a description (hopefully
sufficient to implement a client) of the coaster protocol:
http://wiki.cogkit.org/wiki/Coaster_Protocol

I wrote it because it occurred to me that the protocol is a lot simpler
than it seems, and the difficulty is in writing an efficient
implementation of it.

I'm currently working on a C/C++ client, but I suspect that there are
people out there that can write C++ code a lot more efficiently than I
do. So if you're there and you want to take a stab at it, please read
and let me know if it needs clarification.

Mihael