From bronevetsky1 at llnl.gov  Tue Mar  4 18:53:12 2014
From: bronevetsky1 at llnl.gov (Bronevetsky, Greg)
Date: Wed, 5 Mar 2014 00:53:12 +0000
Subject: [Swift-user] Swift on MOAB
In-Reply-To: <CANa904mb0fbrANidWeCRkWP_Mu5yEwZ4-voPNTw4v6gBEA04YA@mail.gmail.com>
References: <8635C0D1735D2C4BA6E571FD97486FF172E6CD33@PRDEXMBX-05.the-lab.llnl.gov>
	<CANa904mb0fbrANidWeCRkWP_Mu5yEwZ4-voPNTw4v6gBEA04YA@mail.gmail.com>
Message-ID: <8635C0D1735D2C4BA6E571FD97486FF172E71189@PRDEXMBX-05.the-lab.llnl.gov>

After some poking around I got it to mostly work. One issue is that although Moab accepts PBS scripts it does not implement its commands, such as qsub and qstat. I was able to overcome the first issue by creating a soft link named ?qsub? that points to msub. However, I don?t know what to do about qstat. When I do the same linking trick Swift throws errors, probably because it cannot parse the output. Is there a script that takes the output of showq and formats it qstat style? Thanks!

Greg Bronevetsky
Lawrence Livermore National Lab
(925) 424-5756
bronevetsky at llnl.gov<mailto:bronevetsky at llnl.gov>
http://greg.bronevetsky.com

From: Yadu Nand [mailto:yadudoc1729 at gmail.com]
Sent: Friday, February 28, 2014 5:35 PM
To: Bronevetsky, Greg
Cc: swift-user at ci.uchicago.edu
Subject: Re: [Swift-user] Swift on MOAB

Hi Greg,

I went through this page : https://computing.llnl.gov/tutorials/moab/ which mentions that PBS
submit scripts are compatible with Moab. So the Swift PBS provider should work for you.

If there are cluster specific details that differ, we could patch the existing PBS provider to work
with your cluster.

Thanks,
Yadu


On Wed, Feb 26, 2014 at 12:38 PM, Bronevetsky, Greg <bronevetsky1 at llnl.gov<mailto:bronevetsky1 at llnl.gov>> wrote:
I?m running on clusters that use MOAB as the main scheduler, with SLURM on each cluster. Jobs need to be submitted to MOAB. Is there a way for Swift to use MOAB as the scheduler? Thanks!

Greg Bronevetsky
Lawrence Livermore National Lab
(925) 424-5756<tel:%28925%29%20424-5756>
bronevetsky at llnl.gov<mailto:bronevetsky at llnl.gov>
http://greg.bronevetsky.com


_______________________________________________
Swift-user mailing list
Swift-user at ci.uchicago.edu<mailto:Swift-user at ci.uchicago.edu>
https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user



--
Yadu Nand B

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20140305/5ad7409d/attachment.html>

From wilde at mcs.anl.gov  Tue Mar  4 19:55:54 2014
From: wilde at mcs.anl.gov (Wilde, Michael J.)
Date: Wed, 5 Mar 2014 01:55:54 +0000
Subject: [Swift-user] Swift on MOAB
In-Reply-To: <8635C0D1735D2C4BA6E571FD97486FF172E71189@PRDEXMBX-05.the-lab.llnl.gov>
References: <8635C0D1735D2C4BA6E571FD97486FF172E6CD33@PRDEXMBX-05.the-lab.llnl.gov>
	<CANa904mb0fbrANidWeCRkWP_Mu5yEwZ4-voPNTw4v6gBEA04YA@mail.gmail.com>
	<8635C0D1735D2C4BA6E571FD97486FF172E71189@PRDEXMBX-05.the-lab.llnl.gov>
Message-ID: <CF3BDB84.4DE54%wilde@anl.gov>

Greg, we've had discussions about making script-generation for local schedulers more template-driven, and (eg for qstat) use local adapter scripts to a common job-status format (typically jobid + completion-status).

We'll see if we can give you a quick temporary solution, or if you want to help out, some advice on how to proceed.

Regards,

- Mike

From: <Bronevetsky>, Greg <bronevetsky1 at llnl.gov<mailto:bronevetsky1 at llnl.gov>>
Date: Tuesday, March 4, 2014 6:53 PM
To: Yadu Nand <yadudoc1729 at gmail.com<mailto:yadudoc1729 at gmail.com>>
Cc: Swift User <swift-user at ci.uchicago.edu<mailto:swift-user at ci.uchicago.edu>>
Subject: Re: [Swift-user] Swift on MOAB

After some poking around I got it to mostly work. One issue is that although Moab accepts PBS scripts it does not implement its commands, such as qsub and qstat. I was able to overcome the first issue by creating a soft link named ?qsub? that points to msub. However, I don?t know what to do about qstat. When I do the same linking trick Swift throws errors, probably because it cannot parse the output. Is there a script that takes the output of showq and formats it qstat style? Thanks!

Greg Bronevetsky
Lawrence Livermore National Lab
(925) 424-5756
bronevetsky at llnl.gov<mailto:bronevetsky at llnl.gov>
http://greg.bronevetsky.com

From: Yadu Nand [mailto:yadudoc1729 at gmail.com]
Sent: Friday, February 28, 2014 5:35 PM
To: Bronevetsky, Greg
Cc: swift-user at ci.uchicago.edu<mailto:swift-user at ci.uchicago.edu>
Subject: Re: [Swift-user] Swift on MOAB

Hi Greg,

I went through this page : https://computing.llnl.gov/tutorials/moab/ which mentions that PBS
submit scripts are compatible with Moab. So the Swift PBS provider should work for you.

If there are cluster specific details that differ, we could patch the existing PBS provider to work
with your cluster.

Thanks,
Yadu


On Wed, Feb 26, 2014 at 12:38 PM, Bronevetsky, Greg <bronevetsky1 at llnl.gov<mailto:bronevetsky1 at llnl.gov>> wrote:
I?m running on clusters that use MOAB as the main scheduler, with SLURM on each cluster. Jobs need to be submitted to MOAB. Is there a way for Swift to use MOAB as the scheduler? Thanks!

Greg Bronevetsky
Lawrence Livermore National Lab
(925) 424-5756<tel:%28925%29%20424-5756>
bronevetsky at llnl.gov<mailto:bronevetsky at llnl.gov>
http://greg.bronevetsky.com


_______________________________________________
Swift-user mailing list
Swift-user at ci.uchicago.edu<mailto:Swift-user at ci.uchicago.edu>
https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user



--
Yadu Nand B

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20140305/707e1436/attachment.html>

From davidkelly at uchicago.edu  Wed Mar  5 16:14:34 2014
From: davidkelly at uchicago.edu (David Kelly)
Date: Wed, 5 Mar 2014 16:14:34 -0600
Subject: [Swift-user] Swift on MOAB
In-Reply-To: <8635C0D1735D2C4BA6E571FD97486FF172E71189@PRDEXMBX-05.the-lab.llnl.gov>
References: <8635C0D1735D2C4BA6E571FD97486FF172E6CD33@PRDEXMBX-05.the-lab.llnl.gov>
	<CANa904mb0fbrANidWeCRkWP_Mu5yEwZ4-voPNTw4v6gBEA04YA@mail.gmail.com>
	<8635C0D1735D2C4BA6E571FD97486FF172E71189@PRDEXMBX-05.the-lab.llnl.gov>
Message-ID: <CA+_+Ey9p0cVjiMLef76SFmYaLH3Pe7WAWbKpnhg_cK6EjgUmkg@mail.gmail.com>

Hi Greg,

I think having a script that translates from showq output to qstat style
output is possible. Could you please send some example showq output from
your cluster? Thanks!

Regards,
David


On Tue, Mar 4, 2014 at 6:53 PM, Bronevetsky, Greg <bronevetsky1 at llnl.gov>wrote:

>  After some poking around I got it to mostly work. One issue is that
> although Moab accepts PBS scripts it does not implement its commands, such
> as qsub and qstat. I was able to overcome the first issue by creating a
> soft link named "qsub" that points to msub. However, I don't know what to
> do about qstat. When I do the same linking trick Swift throws errors,
> probably because it cannot parse the output. Is there a script that takes
> the output of showq and formats it qstat style? Thanks!
>
>
>
> Greg Bronevetsky
>
> Lawrence Livermore National Lab
>
> (925) 424-5756
>
> bronevetsky at llnl.gov
>
> http://greg.bronevetsky.com
>
>
>
> *From:* Yadu Nand [mailto:yadudoc1729 at gmail.com]
> *Sent:* Friday, February 28, 2014 5:35 PM
> *To:* Bronevetsky, Greg
> *Cc:* swift-user at ci.uchicago.edu
> *Subject:* Re: [Swift-user] Swift on MOAB
>
>
>
> Hi Greg,
>
>
>
> I went through this page : https://computing.llnl.gov/tutorials/moab/ which
> mentions that PBS
>
> submit scripts are compatible with Moab. So the Swift PBS provider should
> work for you.
>
>
>
> If there are cluster specific details that differ, we could patch the
> existing PBS provider to work
>
> with your cluster.
>
>
>
> Thanks,
>
> Yadu
>
>
>
>
>
> On Wed, Feb 26, 2014 at 12:38 PM, Bronevetsky, Greg <bronevetsky1 at llnl.gov>
> wrote:
>
> I'm running on clusters that use MOAB as the main scheduler, with SLURM on
> each cluster. Jobs need to be submitted to MOAB. Is there a way for Swift
> to use MOAB as the scheduler? Thanks!
>
>
>
> Greg Bronevetsky
>
> Lawrence Livermore National Lab
>
> (925) 424-5756
>
> bronevetsky at llnl.gov
>
> http://greg.bronevetsky.com
>
>
>
>
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>
>
>
>
>
> --
>
> Yadu Nand B
>
>
>
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20140305/5bf6b359/attachment.html>

From bronevetsky1 at llnl.gov  Wed Mar  5 16:16:26 2014
From: bronevetsky1 at llnl.gov (Bronevetsky, Greg)
Date: Wed, 5 Mar 2014 22:16:26 +0000
Subject: [Swift-user] Swift on MOAB
In-Reply-To: <CA+_+Ey9p0cVjiMLef76SFmYaLH3Pe7WAWbKpnhg_cK6EjgUmkg@mail.gmail.com>
References: <8635C0D1735D2C4BA6E571FD97486FF172E6CD33@PRDEXMBX-05.the-lab.llnl.gov>
	<CANa904mb0fbrANidWeCRkWP_Mu5yEwZ4-voPNTw4v6gBEA04YA@mail.gmail.com>
	<8635C0D1735D2C4BA6E571FD97486FF172E71189@PRDEXMBX-05.the-lab.llnl.gov>
	<CA+_+Ey9p0cVjiMLef76SFmYaLH3Pe7WAWbKpnhg_cK6EjgUmkg@mail.gmail.com>
Message-ID: <8635C0D1735D2C4BA6E571FD97486FF172E71647@PRDEXMBX-05.the-lab.llnl.gov>

Sure, here you go.

Greg Bronevetsky
Lawrence Livermore National Lab
(925) 424-5756
bronevetsky at llnl.gov<mailto:bronevetsky at llnl.gov>
http://greg.bronevetsky.com

active jobs------------------------
JOBID              USERNAME      STATE NODES   REMAINING            STARTTIME

728592                 osei    Running     1    00:17:19  Wed Mar  5 14:12:33
727574                  fu4    Running     4    00:24:20  Tue Mar  4 22:39:34
...

181 active jobs       19733 of 20704 processors in use by local jobs (95.31%)
                      1287 of 1316 nodes active      (97.80%)

eligible jobs----------------------
JOBID              USERNAME      STATE NODES     WCLIMIT            QUEUETIME

725427             samanta1       Idle     4    16:00:00  Mon Mar  3 19:17:44
725415             samanta1       Idle     4    16:00:00  Mon Mar  3 19:17:28
...

214 eligible jobs

blocked jobs-----------------------
JOBID              USERNAME      STATE NODES     WCLIMIT            QUEUETIME

664849               lordi2   UserHold    42    16:00:00  Wed Feb 12 13:01:02
667137               lordi2   UserHold    42    16:00:00  Thu Feb 13 13:08:06
683547                 ong7       Idle    24    12:00:00  Fri Feb 21 15:05:46
...

116 blocked jobs

Total jobs:  511

From: David Kelly [mailto:davidkelly at uchicago.edu]
Sent: Wednesday, March 05, 2014 2:15 PM
To: Bronevetsky, Greg
Cc: Yadu Nand; swift-user at ci.uchicago.edu
Subject: Re: [Swift-user] Swift on MOAB

Hi Greg,

I think having a script that translates from showq output to qstat style output is possible. Could you please send some example showq output from your cluster? Thanks!

Regards,
David

On Tue, Mar 4, 2014 at 6:53 PM, Bronevetsky, Greg <bronevetsky1 at llnl.gov<mailto:bronevetsky1 at llnl.gov>> wrote:
After some poking around I got it to mostly work. One issue is that although Moab accepts PBS scripts it does not implement its commands, such as qsub and qstat. I was able to overcome the first issue by creating a soft link named "qsub" that points to msub. However, I don't know what to do about qstat. When I do the same linking trick Swift throws errors, probably because it cannot parse the output. Is there a script that takes the output of showq and formats it qstat style? Thanks!

Greg Bronevetsky
Lawrence Livermore National Lab
(925) 424-5756<tel:%28925%29%20424-5756>
bronevetsky at llnl.gov<mailto:bronevetsky at llnl.gov>
http://greg.bronevetsky.com

From: Yadu Nand [mailto:yadudoc1729 at gmail.com<mailto:yadudoc1729 at gmail.com>]
Sent: Friday, February 28, 2014 5:35 PM
To: Bronevetsky, Greg
Cc: swift-user at ci.uchicago.edu<mailto:swift-user at ci.uchicago.edu>
Subject: Re: [Swift-user] Swift on MOAB

Hi Greg,

I went through this page : https://computing.llnl.gov/tutorials/moab/ which mentions that PBS
submit scripts are compatible with Moab. So the Swift PBS provider should work for you.

If there are cluster specific details that differ, we could patch the existing PBS provider to work
with your cluster.

Thanks,
Yadu


On Wed, Feb 26, 2014 at 12:38 PM, Bronevetsky, Greg <bronevetsky1 at llnl.gov<mailto:bronevetsky1 at llnl.gov>> wrote:
I'm running on clusters that use MOAB as the main scheduler, with SLURM on each cluster. Jobs need to be submitted to MOAB. Is there a way for Swift to use MOAB as the scheduler? Thanks!

Greg Bronevetsky
Lawrence Livermore National Lab
(925) 424-5756<tel:%28925%29%20424-5756>
bronevetsky at llnl.gov<mailto:bronevetsky at llnl.gov>
http://greg.bronevetsky.com


_______________________________________________
Swift-user mailing list
Swift-user at ci.uchicago.edu<mailto:Swift-user at ci.uchicago.edu>
https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user



--
Yadu Nand B


_______________________________________________
Swift-user mailing list
Swift-user at ci.uchicago.edu<mailto:Swift-user at ci.uchicago.edu>
https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20140305/da997326/attachment.html>

From iraicu at cs.iit.edu  Mon Mar 24 12:48:13 2014
From: iraicu at cs.iit.edu (Ioan Raicu)
Date: Mon, 24 Mar 2014 12:48:13 -0500
Subject: [Swift-user] Call for Posters at ACM HPDC 2014
Message-ID: <53306FDD.10906@cs.iit.edu>

Call for Posters
Submission Deadline: May 16 2014
http://www.hpdc.org/2014/posters/call-for-posters/

HPDC'14 will feature a poster session that will provide the right environment for lively and informal discussions on various high performance parallel and distributed computing topics.
The poster session will be held on Wednesday, June 25, in the late afternoon.

Participating posters will be selected based on the following criteria:

- Submissions must describe new, interesting ideas on any HPDC topics of interest
- Submissions can present work in progress, and we strongly encourage the authors to include preliminary experimental results, if available
- Student submissions meeting the above criteria will be given preference

We invite all potential authors to submit their contribution to this poster session in the form of a two-page PDF abstract (we recommend using the ACM Proceedings style, and fonts not smaller than 10 point).
Please provide the following information in your PDF file:
- Poster title
- Author names, affiliations, and email addresses
- Note which authors, if any, are students

Abstracts must be submitted through email to chandra AT cs DOT umn DOT edu before May 16 2014, 5:00pm EDT.
Authors will be notified of acceptance or rejection via e-mail by May 23, 2014. No reviews will be provided.
Accepted posters will be published online on the conference website. Details about the poster presentation (e.g., poster size) will be available closer to the conference.

For any questions about the submission, selection, and presentation of the accepted posters, please contact the Posters Chair, Abhishek Chandra, University of Minnesota (email: chandra AT cs DOT umn DOT edu).

-- 
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor, Illinois Institute of Technology (IIT)
Guest Research Faculty, Argonne National Laboratory (ANL)
=================================================================
Data-Intensive Distributed Systems Laboratory, CS/IIT
Distributed Systems Laboratory, MCS/ANL
=================================================================
Editor: IEEE TCC, Springer Cluster, Springer JoCCASA
Chair:  IEEE/ACM MTAGS, ACM ScienceCloud
=================================================================
Cel:      1-847-722-0876
Office:   1-312-567-5704
Email:    iraicu at cs.iit.edu
Web:      http://www.cs.iit.edu/~iraicu/
Web:      http://datasys.cs.iit.edu/
LinkedIn: http://www.linkedin.com/in/ioanraicu
Google:   http://scholar.google.com/citations?user=jE73HYAAAAAJ
=================================================================
=================================================================

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20140324/f7935ad5/attachment.html>

From iraicu at cs.iit.edu  Wed Mar 26 09:07:27 2014
From: iraicu at cs.iit.edu (Ioan Raicu)
Date: Wed, 26 Mar 2014 09:07:27 -0500
Subject: [Swift-user] Call for Participation: IEEE/ACM CCGrid 2014 in
 Chicago May 26-29 -- Early Bird Registration due April 15th
Message-ID: <5332DF1F.1050404@cs.iit.edu>

------------------------- Call for Participation -------------------------

-------------------------- IEEE/ACM CCGrid 2014 --------------------------

14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing

May 26-29, 2014

Chicago, IL, USA

http://datasys.cs.iit.edu/events/CCGrid2014/

Upcoming Important Dates:

Early Bird Registration: April 15

Registration: May 5

Room Reservation: May 5

Workshops: May 26

Conference: May 27-29

Tutorial: May 29

A Message from the CCGrid 2014 General Chairs:

It is our great pleasure to welcome you to the 14th IEEE/ACM 
International Symposium on Cluster, Cloud and Grid Computing (CCGrid 
2014) and to the great global city of Chicago. CCGrid is a forum for all 
distributed computing technologies and for all technology stakeholders. 
The inaugural CCGrid conference was held in Brisbane, Australia, in 
2001. Three years later, in 2004, it was held in Chicago. Ten years have 
passed since then. In 2004, Chicago was a leader in Grid computing. 
Today, it is a center of Cloud technology innovations. CCGrid 2014 is 
just like the city of Chicago, having naturally blended and mixed with 
the historical glories and modern stallers into its unique identity. 
Come and join us to meet preeminent computer scientists, leading 
creators of Grid and Cluster technologies, and new shining stars, the 
significant contributors of today's Cloud and big data management 
advances. We promise you a memorable experience unequaled at other 
conferences.

Late May is the perfect time to visit Chicago. Flowers are booming, 
fountains are sprinkling, and sail boats are in the bay; but most of the 
summer tours are not in yet. The conference hotel, the Hyatt Regency 
Chicago, is in an ideal location. It is one block off The Magnificent 
Mile and near Millennium Park, Lake Michigan, and the Chicago River. 
Even you do not do shopping, simply walking along Michigan Ave or the 
Chicago River, in the day time or in the evening, is a pleasure. Plus 
the special hotel rate for CCGrid attendees is very attractive. The 
conference banquet will be held on a lake cruise in the evening of 
Wednesday. That night Chicago will have a half hour long firework show 
on the lake. Of course, the most exciting part is the outstanding 
programs offered by CCGrid 2014. From Keynotes to Technical Papers, 
CCGrid14 has its first class programs in all categories. We heartily 
appreciate the committee members and volunteers who have put the 
wonderful programs together.

CCGrid 2014 features keynote talks, tutorials, workshops, poster 
sessions and demos, competition, student travel awards, panel, as well 
technical papers,. We are pleased to have This year the prestigious IEEE 
Medal for Excellence in Scalable Computing award winner, Professor Yves 
Robert at Ecole Normale Superieure de Lyon, is also a keynote speaker of 
CCGrid 2014. Prof Robert is authority in algorithm design and analysis.

CCGrid 2014 received 302 paper submissions from 40 countries. After 
administrative filtering, 283 papers received full reviews. In total, 
1089 reviews were conducted and 54 papers were accepted, with an 
acceptance rate of 19% (54/283). There are eight workshops on Monday, 
May 26, 2014, and five concurrent tutorials in the afternoon of 
Thursday, May 29. Eight posters and two demos are selected for the 
poster session in the evening of Tuesday, May 27, and ten papers are 
accepted for the Doctoral Symposium program on Wednesday, May 28. The 
IEEE SCALE Dr. Farnam Jahanian, the National Science Foundation 
Assistant Director for the Computer and Information Science and 
Engineering (CISE), to deliver the opening keynote. The keynote speaker 
of the second day, Prof. Ion Stoica, is a professor of Computer Science 
at University of California at Berkeley, and is known by his current 
research projects, Mesos and Spark. SCALE Challenge competition will be 
held live, with finalists judged by their demonstrations on Tuesday and 
the winner announced on Thursday. CCGrid 2014 has received generous 
sponsorship from the U.S. National Science Foundation and the IEEE 
Technical Committee on Scalable Computing to assist 17 students to 
attend this conference. A greeting and round-table session are also 
arranged for the student awardees.

The success of CCGrid 2014 is due to the dedicated efforts and high 
standards of numerous international volunteers. Our long thank you list 
starts with the two excellent Program Chairs: Kirk W. Cameron and 
Dimitrios S. Nikolopoulos, and the Program Committee. Special thanks to 
the Program Committee Area Chairs who braved the coldest Chicago day in 
decades to run the program committee meeting on January 27, 2014. We 
thank Workshops Co-Chairs Zhiling Lan and Matei Ripeanu, the chairs and 
PC committees of the various workshops, and our Publicity Chairs for 
getting the word out about the conference. We thank Tutorials Co-Chairs 
Kate Keahey and Radu Prodan, Poster and Research Demo Co-Chairs Borja 
Sotomayor and Hui Jin, Doctoral Symposium Chair Judy Qiu, the Student 
Awards Chair Yong Chen, and the SCALE Challenge Coordinator Douglas 
Thain. The Cyber Co-chairs, Ge Rong and Wei Tang, did a wonderful job 
with the conference website. Dr. Pavan Balaji, the Proceedings Chair, 
ensured the publication of the conference proceedings. We are especially 
grateful to the Local Organizing Chairs, Ioan Raicu and Kyle Chard, who 
did a tremendous job on innumerable tasks, from identifying the hotel to 
negotiating the price of the banquet. Thanks are also due to our 
sponsors, namely, IEEE, ACM, TCSC, and the organizational supporters at 
Illinois Institute of Technology and the University of Chicago. 
Ultimately, however, the success of the conference will be judged by the 
attendees' experience. We hope that the conference will provide you with 
a valuable opportunity to share ideas, communicate, learn, and network.

We wish everyone a successful, stimulating, and rewarding meeting and 
look forward to seeing you again at future CCGrid conferences.

Ian Foster (University of Chicago and Argonne National Laboratory)

Xian-He Sun (Illinois Institute of Technology)

-- 
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor, Illinois Institute of Technology (IIT)
Guest Research Faculty, Argonne National Laboratory (ANL)
=================================================================
Data-Intensive Distributed Systems Laboratory, CS/IIT
Distributed Systems Laboratory, MCS/ANL
=================================================================
Editor: IEEE TCC, Springer Cluster, Springer JoCCASA
Chair:  IEEE/ACM MTAGS, ACM ScienceCloud
=================================================================
Cel:      1-847-722-0876
Office:   1-312-567-5704
Email:    iraicu at cs.iit.edu
Web:      http://www.cs.iit.edu/~iraicu/
Web:      http://datasys.cs.iit.edu/
LinkedIn: http://www.linkedin.com/in/ioanraicu
Google:   http://scholar.google.com/citations?user=jE73HYAAAAAJ
=================================================================
=================================================================

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20140326/e2c3f732/attachment.html>

From wilde at anl.gov  Thu Mar 27 22:00:35 2014
From: wilde at anl.gov (Michael Wilde)
Date: Thu, 27 Mar 2014 22:00:35 -0500
Subject: [Swift-user] [Swift-devel] Multiple output files
In-Reply-To: <B38ADDB4-5F5B-4D81-948C-68665AED9247@uchicago.edu>
References: <FCF28DE0-9D06-434C-98DA-6286BFA80D09@uchicago.edu>	<532A305B.4000600@anl.gov>	<A9524469-E43E-4021-AFBB-108BA6E96435@uchicago.edu>	<532B9B8C.7000007@anl.gov>	<3FD5A438-74FE-418C-822D-229125EA6CDD@uchicago.edu>
	<532C3B7C.30609@anl.gov> <532C410C.5080309@anl.gov>
	<532DBAF8.9030300@anl.gov>
	<B38ADDB4-5F5B-4D81-948C-68665AED9247@uchicago.edu>
Message-ID: <5334E5D3.1020309@anl.gov>

Jonathan, I'd need to test this, but my suspicion is that specifying 
location only changes the directory that the mapper should look in.

But you also need to give it a prefix and/or a suffix - else it won't 
know what you want it to map *within* that drectory.

My guess is that you had a mapper with "prefix=" that worked, and you 
just changed "prefix" to "location"?

Also - please see Mihael's post to swift-devel in the past hour. He has 
committed the enhancement to dynamically map an output array of files on 
the compute node, as you requested and we discussed.  I think you'll 
have to build a swift from trunk source to try this, but thats only an 
svn co and a few commands (see Download at swift-lang.org).

-- 
Mike

On 3/27/14, 5:02 PM, Jonathan Ozik wrote:
> Mike,
>
> I've been making a good amount of progress and I'm able to 
> successfully run a number of concurrent instances locally.
>
> I had a question about the filesys_mapper, and how it might (or might 
> not) be able to handle nested folders. If I point it to a "location" 
> like this:
> file data[] <filesys_mapper;location="data">;
> I get a file not found error when it is used as an input (presumably 
> because that is when the mapper gets resolved) if there is a nested 
> folder within that location.
> data/
> temp.txt
> insideData/
> temp1.txt
>
> Is this expected behavior? If so, is there a workaround to gather all 
> files under a folder as input?
>
> Jonathan
>
> On Mar 22, 2014, at 11:31 AM, Michael Wilde <wilde at anl.gov 
> <mailto:wilde at anl.gov>> wrote:
>
>> This is now filed as enhancement bug 1225 and assigned to you, 
>> Mihael.  The description has been revised as you suggested, to 
>> propose that the feature be first done using simple_mapper rather 
>> than a new mapper.
>>
>> - Mike
>>
>> On 3/21/14, 8:39 AM, Michael Wilde wrote:
>>> Mihael, All,
>>>
>>> I'd like to propose a Swift/K feature to provide a reasonable 
>>> solution to this very common need for an app to return a dynamically 
>>> determined set of files.
>>>
>>> file dynarry[ ] <runtime; prefix="myoutput.", suffix=".out", 
>>> indexes="int">;
>>>
>>> dynarry = myApp(myArgs);
>>>
>>> The "runtime" mapper should initially have the same arguments and 
>>> semantics (roughly) as simple_mapper, except for two new arguments:
>>>
>>>  "indexes" which determines how the matched file names will be 
>>> indexed in the returned array
>>> "int" | "string" | "sequential"
>>> sequential: return the matched files as consecutive integer indices 
>>> starting with 0
>>> int: expect the filename component between prefix and suffix to be 
>>> convertible to an integer, and use that as the index
>>> eg myfile.012.out and myfile.204.out will return an array with the 
>>> mapped files at indices 12 and 204.
>>> string: similar to int but return a string-indexed associative array.
>>> "sequential" is simplest and should be the default.
>>>
>>> "paths" which determines if the match names will be absolute or 
>>> relative to the job dir
>>> paths="relative" | "absolute"
>>> (may not be needed if this can be determined uniquely based on the 
>>> location argument.
>>>
>>> swiftwrap will allow array variables mapped in this manner to have 
>>> any number of files, including zero. I.e. "runtime-mapped" files 
>>> should not be listed in the expected output list for an app 
>>> invocation. Its up to the users app to ensure that some files match 
>>> the pattern. An additional arg could set e.g. minfiles and/or 
>>> maxfiles, in which case the wrapper code needs to validate the count 
>>> of files matched and returned, but not their exact names.
>>>
>>> We can call this mapper "experimental" until we validate its 
>>> usability and suitability as a permanent feature. But as we hope to 
>>> revise the entire mapper family and semantics, in a sense all 
>>> mappers are subject to change.
>>>
>>> Mihael, is the definition sound, and how long would it take you to 
>>> develop it?
>>>
>>> Thanks,
>>>
>>> - Mike
>>>
>>
>> -- 
>> Michael Wilde
>> Mathematics and Computer Science          Computation Institute
>> Argonne National Laboratory               The University of Chicago
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20140327/d55a7b35/attachment.html>