From yadudoc1729 at gmail.com  Sat Jun  1 14:37:23 2013
From: yadudoc1729 at gmail.com (Yadu Nand)
Date: Sat, 1 Jun 2013 14:37:23 -0500
Subject: [Swift-devel] Coaster provider staging timeout problems
	lingering
In-Reply-To: <913312318.1540830.1370017776370.JavaMail.root@mcs.anl.gov>
References: <1716155600.154516.1363096261076.JavaMail.root@mcs.anl.gov>
	<913312318.1540830.1370017776370.JavaMail.root@mcs.anl.gov>
Message-ID: <CANa904=u7kTQknJXMMhYmKS2H4w7O4Vpx5ueztKGqv6qX9kt6g@mail.gmail.com>

Mike,

Yes!

I see the same error string in my logs as well:
org.globus.cog.karajan.workflow.service.TimeoutException: Channel
timed out. lastTime=130601-180549.457, now=130601-180750.058,
channel=GSSSChannel-09\
20050760(1)[service-60881]
        at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.checkTimeouts(AbstractKarajanChannel.java:130)
        at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel$1.run(AbstractKarajanChannel.java:121)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)

What I find interesting is the way this bug goes silent with fewer app
invocations.
When I tried to request just one worker on Beagle with debug error
logging, I think there wouldn't be a need to stage files in parallel
and I do not see any timeouts. All these details are in the comments
for Bug: 1006 (https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1006)

Mihael, I hope these logs and tests help. Let me know if you need anything more.

-Yadu

On Fri, May 31, 2013 at 11:29 AM, Michael Wilde <wilde at mcs.anl.gov> wrote:
> I have heard from David a while back and Yadu this week that coaster provider staging timeouts remain a problem, albeit a far less frequent one.
>
> Yadu seems to have a test (which he is polishing) that can reproduce the problem readily.
>
> Yadu, do the coaster provider staging timeouts that you are seeing on the ex-search app match what I encountered back in March on OSG via UC3 (below)?
>
> Please either file a ticket asap and assign to Mihael or locate an existing ticket for this bug and add more logs and incidents to it.
>
> David, can you tell us what your impression of remaining provider staging failure scenarios are, ideally via an existing ticket if there's one open for this?
>
> Thanks,
>
> - Mike
>
> ----- Forwarded Message -----
> From: "Michael Wilde" <wilde at mcs.anl.gov>
> To: "Mihael Hategan" <hategan at mcs.anl.gov>
> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> Sent: Tuesday, March 12, 2013 8:51:01 AM
> Subject: [Swift-devel] Coaster run to UC3 dies with channel timeout
>
> This demo (for OSG all-hands) was running fairly reliably, 100's to a few thousand 30-second tasks to UC3 with flocking to OSG and other pools.
>
> But just got a failure, so it looks like sporadic problems remain.
>
> Running Swift 0.94 latest rev.
>
> Log is on midway in:
>
> /home/wilde/osgdemo/modis/svn/swiftdemo/test.uc3
> -rw-rw-r-- 1 wilde wilde 11632001 Mar 12 08:42 saved/modis-20130312-1335-p30ylps9.log
>
> I'll file a ticket once we get a sense of the frequency.
>
> - Mike
>
>
>
>  Progress:  time: Tue, 12 Mar 2013 13:42:28 +0000  Selecting site:461  Stage in:10  Submitted:782  Active:204  Stage out:4  Finished successfully:1539
> Progress:  time: Tue, 12 Mar 2013 13:42:29 +0000  Selecting site:453  Stage in:6  Submitted:779  Active:215  Finished successfully:1547
> Progress:  time: Tue, 12 Mar 2013 13:42:30 +0000  Selecting site:439  Stage in:16  Submitting:1  Submitted:776  Active:204  Stage out:2  Finished successfully:1562
> Execution failed:
>         Exception in perl:
>     Arguments: [getlanduse.pl, input/h06v33.rgb]
>     Host: uc3
>     Directory: modis-20130312-1335-p30ylps9/jobs/7/perl-7s7qvh6l
>
> Caused by:
>         Task failed: null
> org.globus.cog.karajan.workflow.service.TimeoutException: Channel timed out. lastTime=130312-084030.762, now=130312-084231.763, channel=TCP-0312-3508510-000259-000000
>         at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.checkTimeouts(AbstractKarajanChannel.java:131)
>         at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel$1.run(AbstractKarajanChannel.java:122)
>         at java.util.TimerThread.mainLoop(Timer.java:555)
>         at java.util.TimerThread.run(Timer.java:505)
>
>         getLandUse, modis.swift, line 24
> swift$ pwd
> /home/wilde/osgdemo/modis/svn/swiftdemo/test.uc3
> swift$ ls
> cf              input/    modis-20130312-1335-p30ylps9.0.rlog  modis-20130312-1335-p30ylps9.log  saved/     tc
> getlanduse.pl*  landuse/  modis-20130312-1335-p30ylps9.d/      run*                              swift.log  uc3.xml
> swift$ e ../save
> swift$ save
> swift$ ls saved
> modis-20130312-1326-n9rofj6e.d/   modis-20130312-1329-f2a2eic4.log     modis-20130312-1335-p30ylps9.log
> modis-20130312-1326-n9rofj6e.log  modis-20130312-1335-p30ylps9.0.rlog  swift.log
> modis-20130312-1329-f2a2eic4.d/   modis-20130312-1335-p30ylps9.d/
> swift$ ls saved/modis-20130312-1335-p30ylps9.log
> saved/modis-20130312-1335-p30ylps9.log
> swift$ pwd; ls -l saved/modis-20130312-1335-p30ylps9.log
> /home/wilde/osgdemo/modis/svn/swiftdemo/test.uc3
> -rw-rw-r-- 1 wilde wilde 11632001 Mar 12 08:42 saved/modis-20130312-1335-p30ylps9.log
> swift$
>
>
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel



-- 
Yadu Nand B


From zhaozhang at uchicago.edu  Sun Jun  2 14:12:30 2013
From: zhaozhang at uchicago.edu (ZHAO ZHANG)
Date: Sun, 2 Jun 2013 12:12:30 -0700
Subject: [Swift-devel] Iterative PageRank in Swift
Message-ID: <6F85720E-81C4-42BD-B12F-6DF281DCE023@uchicago.edu>

Dear all,

I have been working with my cousin on an iterative PageRank implementation with Swift for his graduation project. We now encounter an problem: we try to use "file fn[]" as intermediate data between two stages, however, it does not work well. 

The app and stage definition looks like below:
zhaozhang at bigben:/var/tmp/workplace$ cat PageRank-new.swift
type file; 

app (file t) distribution (file f, file s) {   
    distribution @filename(f) @filename(t) @filename(s);
}

app (file t[]) partition (file f) {   
    partition @filename(f) "16";
}

app (file t) aggregation (file f[]){
    aggregation @filename(t) @filenames(f);
}

app (file t) cat (file f[]){
    cat @filenames(f) stdout=@filename(t);
}

app (file t) sort (file f){
    sort "-nrk 2" @filename(f) stdout=@filename(t);
}

(file fn[])map(file input[], file score){
   foreach f,i in input {
      file c<regexp_mapper;
         source=@f,
         match="input/(.*)",
         transform="temp/\\1">;
      c = distribution(f, score);  
      fn[i] = c;
   }
}

(file matrix[][])shuffle(file fn[]){
   foreach c, j in fn{
	file output[] <ext; exec="bin/mapper.sh", source=@filename(c), scale=16>;
	output = partition(c);
	foreach f, k in output{
		matrix[k][j] = output[k];
	}
   }
}

(file final)reduce(file matrix[][]){
   file result[];	  
   foreach fl, k in matrix{
      file output <single_file_mapper; file=@strcat("result/result-", @toString(k))>;
      output = aggregation(fl);
      result[k] = output;
   }   

   final = cat(result);
}


If I write the main function as below, it does not work: it seems the intermediate files are not mapped to the expected file names.

//below are main function
file input[] <filesys_mapper; location="input", prefix="links-">;
file matrix[][];
file fn[];

int loop=0;
file score <single_file_mapper; file=@strcat("score.txt.", @toString(loop))>;
file final <single_file_mapper;file=@strcat("score.txt.", @toString(loop+1))>;
file sorted <single_file_mapper;file=@strcat("score.txt.", @toString(loop+1), ".sorted")>;

fn = map(input, score);
matrix = shuffle(fn);
final = reduce(matrix);
sorted = sort(final);

The execution failed with the following message:
Swift 0.94 swift-r6492 cog-r3658

RunID: 20130602-1348-yresjj56
Progress:  time: Sun, 02 Jun 2013 13:48:49 -0500
Progress:  time: Sun, 02 Jun 2013 13:48:51 -0500  Selecting site:3  Checking status:1
Progress:  time: Sun, 02 Jun 2013 13:48:52 -0500  Selecting site:3  Checking status:1  Finished successfully:2
Progress:  time: Sun, 02 Jun 2013 13:48:53 -0500  Selecting site:3  Checking status:1  Finished successfully:4
Execution failed:
	Exception in partition:
    Arguments: [temp/links-part-0001, 16]
    Host: localhost
    Directory: PageRank-new-20130602-1348-yresjj56/jobs/i/partition-ishh5dal
    stderr.txt: 
    stdout.txt: 
Caused by:
	The following output files were not created by the application: _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-0, _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-1, _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-2, _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-3, _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-4, _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-5, _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-6, _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-7, _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-8, _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-9, _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-10, _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-11, _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-12, _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-13, _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-14, _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-15
	partition, PageRank-new.swift, line 38
	shuffle, PageRank-new.swift, line 88



However, if I put the stages in Iterate control struct: it works.

//below are main function
file input[] <filesys_mapper; location="input", prefix="links-">;
file matrix[][];
file fn[];

/*iterate loop{
   iterate i{
      if (i==0){
         file score <single_file_mapper; file=@strcat("score.txt.", @toString(loop))>;
         fn = map(input, score);
      }
      if(i==1){
         matrix = shuffle(fn);
      }
      if(i==2){
         file final <single_file_mapper;file=@strcat("score.txt.", @toString(loop+1))>;
         final = reduce(matrix);
         file sorted <single_file_mapper;file=@strcat("score.txt.", @toString(loop+1), ".sorted")>;
         sorted = sort(final);
      }
   }until(i==3);
}until(loop==1);*/


I also checked SwiftMontage implementation, it was also written in this way, so I assumed the first draft should work some time ago. Is this a already known problem?

Best
Zhao



From wilde at mcs.anl.gov  Sun Jun  2 14:48:58 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Sun, 2 Jun 2013 14:48:58 -0500 (CDT)
Subject: [Swift-devel] Iterative PageRank in Swift
In-Reply-To: <1082050612.1938203.1370202514560.JavaMail.root@mcs.anl.gov>
Message-ID: <528412417.1939026.1370202538222.JavaMail.root@mcs.anl.gov>

Zhao, Im studying this.  Can you post a copy of mapper.sh?

Can you put a copy on a local machine here (both failing and working version) that I can experiment with?

Thanks,

- Mike


----- Original Message -----
> From: "ZHAO ZHANG" <zhaozhang at uchicago.edu>
> To: "Swift Devel" <swift-devel at ci.uchicago.edu>
> Sent: Sunday, June 2, 2013 2:12:30 PM
> Subject: [Swift-devel] Iterative PageRank in Swift
> 
> Dear all,
> 
> I have been working with my cousin on an iterative PageRank
> implementation with Swift for his graduation project. We now
> encounter an problem: we try to use "file fn[]" as intermediate data
> between two stages, however, it does not work well.
> 
> The app and stage definition looks like below:
> zhaozhang at bigben:/var/tmp/workplace$ cat PageRank-new.swift
> type file;
> 
> app (file t) distribution (file f, file s) {
>     distribution @filename(f) @filename(t) @filename(s);
> }
> 
> app (file t[]) partition (file f) {
>     partition @filename(f) "16";
> }
> 
> app (file t) aggregation (file f[]){
>     aggregation @filename(t) @filenames(f);
> }
> 
> app (file t) cat (file f[]){
>     cat @filenames(f) stdout=@filename(t);
> }
> 
> app (file t) sort (file f){
>     sort "-nrk 2" @filename(f) stdout=@filename(t);
> }
> 
> (file fn[])map(file input[], file score){
>    foreach f,i in input {
>       file c<regexp_mapper;
>          source=@f,
>          match="input/(.*)",
>          transform="temp/\\1">;
>       c = distribution(f, score);
>       fn[i] = c;
>    }
> }
> 
> (file matrix[][])shuffle(file fn[]){
>    foreach c, j in fn{
> 	file output[] <ext; exec="bin/mapper.sh", source=@filename(c),
> 	scale=16>;
> 	output = partition(c);
> 	foreach f, k in output{
> 		matrix[k][j] = output[k];
> 	}
>    }
> }
> 
> (file final)reduce(file matrix[][]){
>    file result[];
>    foreach fl, k in matrix{
>       file output <single_file_mapper; file=@strcat("result/result-",
>       @toString(k))>;
>       output = aggregation(fl);
>       result[k] = output;
>    }
> 
>    final = cat(result);
> }
> 
> 
> If I write the main function as below, it does not work: it seems the
> intermediate files are not mapped to the expected file names.
> 
> //below are main function
> file input[] <filesys_mapper; location="input", prefix="links-">;
> file matrix[][];
> file fn[];
> 
> int loop=0;
> file score <single_file_mapper; file=@strcat("score.txt.",
> @toString(loop))>;
> file final <single_file_mapper;file=@strcat("score.txt.",
> @toString(loop+1))>;
> file sorted <single_file_mapper;file=@strcat("score.txt.",
> @toString(loop+1), ".sorted")>;
> 
> fn = map(input, score);
> matrix = shuffle(fn);
> final = reduce(matrix);
> sorted = sort(final);
> 
> The execution failed with the following message:
> Swift 0.94 swift-r6492 cog-r3658
> 
> RunID: 20130602-1348-yresjj56
> Progress:  time: Sun, 02 Jun 2013 13:48:49 -0500
> Progress:  time: Sun, 02 Jun 2013 13:48:51 -0500  Selecting site:3
>  Checking status:1
> Progress:  time: Sun, 02 Jun 2013 13:48:52 -0500  Selecting site:3
>  Checking status:1  Finished successfully:2
> Progress:  time: Sun, 02 Jun 2013 13:48:53 -0500  Selecting site:3
>  Checking status:1  Finished successfully:4
> Execution failed:
> 	Exception in partition:
>     Arguments: [temp/links-part-0001, 16]
>     Host: localhost
>     Directory:
>     PageRank-new-20130602-1348-yresjj56/jobs/i/partition-ishh5dal
>     stderr.txt:
>     stdout.txt:
> Caused by:
> 	The following output files were not created by the application:
> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-0,
> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-1,
> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-2,
> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-3,
> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-4,
> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-5,
> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-6,
> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-7,
> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-8,
> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-9,
> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-10,
> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-11,
> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-12,
> 	_concurrent/fn-139240b8-8
>  1cc-4b22-8088-aa5aedd98afe--array//elt-3-13,
>  _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-14,
>  _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-15
> 	partition, PageRank-new.swift, line 38
> 	shuffle, PageRank-new.swift, line 88
> 
> 
> 
> However, if I put the stages in Iterate control struct: it works.
> 
> //below are main function
> file input[] <filesys_mapper; location="input", prefix="links-">;
> file matrix[][];
> file fn[];
> 
> /*iterate loop{
>    iterate i{
>       if (i==0){
>          file score <single_file_mapper; file=@strcat("score.txt.",
>          @toString(loop))>;
>          fn = map(input, score);
>       }
>       if(i==1){
>          matrix = shuffle(fn);
>       }
>       if(i==2){
>          file final <single_file_mapper;file=@strcat("score.txt.",
>          @toString(loop+1))>;
>          final = reduce(matrix);
>          file sorted <single_file_mapper;file=@strcat("score.txt.",
>          @toString(loop+1), ".sorted")>;
>          sorted = sort(final);
>       }
>    }until(i==3);
> }until(loop==1);*/
> 
> 
> I also checked SwiftMontage implementation, it was also written in
> this way, so I assumed the first draft should work some time ago. Is
> this a already known problem?
> 
> Best
> Zhao
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 


From wilde at mcs.anl.gov  Sun Jun  2 15:21:24 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Sun, 2 Jun 2013 15:21:24 -0500 (CDT)
Subject: [Swift-devel] Iterative PageRank in Swift
In-Reply-To: <528412417.1939026.1370202538222.JavaMail.root@mcs.anl.gov>
Message-ID: <1428011530.1945766.1370204484519.JavaMail.root@mcs.anl.gov>

Zhao,

The immediate failure in the run below seems to be due to partition() not creating the following 16 files below its *work* directory:

_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-0
_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-1
...
_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-15

Im assuming mapper.sh gets the name of an element of fn[], which has been mapped by the concurrent mapper, and returns an array mapping of the original name suffixed by -0 through -16?

Is that the expected behavior?

Is the partition() app coded to do this, including making the parent directories ./_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array/ ?

I cant yet explain why/how this worked when you serialized the main (open code) function, but there's enough going on in this code that I'd carefully check the behavior of each stage.

- Mike

----- Original Message -----
> From: "Michael Wilde" <wilde at mcs.anl.gov>
> To: "ZHAO ZHANG" <zhaozhang at uchicago.edu>
> Cc: "swift-devel" <swift-devel at ci.uchicago.edu>
> Sent: Sunday, June 2, 2013 2:48:58 PM
> Subject: Re: [Swift-devel] Iterative PageRank in Swift
> 
> Zhao, Im studying this.  Can you post a copy of mapper.sh?
> 
> Can you put a copy on a local machine here (both failing and working
> version) that I can experiment with?
> 
> Thanks,
> 
> - Mike
> 
> 
> ----- Original Message -----
> > From: "ZHAO ZHANG" <zhaozhang at uchicago.edu>
> > To: "Swift Devel" <swift-devel at ci.uchicago.edu>
> > Sent: Sunday, June 2, 2013 2:12:30 PM
> > Subject: [Swift-devel] Iterative PageRank in Swift
> > 
> > Dear all,
> > 
> > I have been working with my cousin on an iterative PageRank
> > implementation with Swift for his graduation project. We now
> > encounter an problem: we try to use "file fn[]" as intermediate
> > data
> > between two stages, however, it does not work well.
> > 
> > The app and stage definition looks like below:
> > zhaozhang at bigben:/var/tmp/workplace$ cat PageRank-new.swift
> > type file;
> > 
> > app (file t) distribution (file f, file s) {
> >     distribution @filename(f) @filename(t) @filename(s);
> > }
> > 
> > app (file t[]) partition (file f) {
> >     partition @filename(f) "16";
> > }
> > 
> > app (file t) aggregation (file f[]){
> >     aggregation @filename(t) @filenames(f);
> > }
> > 
> > app (file t) cat (file f[]){
> >     cat @filenames(f) stdout=@filename(t);
> > }
> > 
> > app (file t) sort (file f){
> >     sort "-nrk 2" @filename(f) stdout=@filename(t);
> > }
> > 
> > (file fn[])map(file input[], file score){
> >    foreach f,i in input {
> >       file c<regexp_mapper;
> >          source=@f,
> >          match="input/(.*)",
> >          transform="temp/\\1">;
> >       c = distribution(f, score);
> >       fn[i] = c;
> >    }
> > }
> > 
> > (file matrix[][])shuffle(file fn[]){
> >    foreach c, j in fn{
> > 	file output[] <ext; exec="bin/mapper.sh", source=@filename(c),
> > 	scale=16>;
> > 	output = partition(c);
> > 	foreach f, k in output{
> > 		matrix[k][j] = output[k];
> > 	}
> >    }
> > }
> > 
> > (file final)reduce(file matrix[][]){
> >    file result[];
> >    foreach fl, k in matrix{
> >       file output <single_file_mapper;
> >       file=@strcat("result/result-",
> >       @toString(k))>;
> >       output = aggregation(fl);
> >       result[k] = output;
> >    }
> > 
> >    final = cat(result);
> > }
> > 
> > 
> > If I write the main function as below, it does not work: it seems
> > the
> > intermediate files are not mapped to the expected file names.
> > 
> > //below are main function
> > file input[] <filesys_mapper; location="input", prefix="links-">;
> > file matrix[][];
> > file fn[];
> > 
> > int loop=0;
> > file score <single_file_mapper; file=@strcat("score.txt.",
> > @toString(loop))>;
> > file final <single_file_mapper;file=@strcat("score.txt.",
> > @toString(loop+1))>;
> > file sorted <single_file_mapper;file=@strcat("score.txt.",
> > @toString(loop+1), ".sorted")>;
> > 
> > fn = map(input, score);
> > matrix = shuffle(fn);
> > final = reduce(matrix);
> > sorted = sort(final);
> > 
> > The execution failed with the following message:
> > Swift 0.94 swift-r6492 cog-r3658
> > 
> > RunID: 20130602-1348-yresjj56
> > Progress:  time: Sun, 02 Jun 2013 13:48:49 -0500
> > Progress:  time: Sun, 02 Jun 2013 13:48:51 -0500  Selecting site:3
> >  Checking status:1
> > Progress:  time: Sun, 02 Jun 2013 13:48:52 -0500  Selecting site:3
> >  Checking status:1  Finished successfully:2
> > Progress:  time: Sun, 02 Jun 2013 13:48:53 -0500  Selecting site:3
> >  Checking status:1  Finished successfully:4
> > Execution failed:
> > 	Exception in partition:
> >     Arguments: [temp/links-part-0001, 16]
> >     Host: localhost
> >     Directory:
> >     PageRank-new-20130602-1348-yresjj56/jobs/i/partition-ishh5dal
> >     stderr.txt:
> >     stdout.txt:
> > Caused by:
> > 	The following output files were not created by the application:
> > 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-0,
> > 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-1,
> > 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-2,
> > 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-3,
> > 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-4,
> > 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-5,
> > 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-6,
> > 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-7,
> > 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-8,
> > 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-9,
> > 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-10,
> > 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-11,
> > 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-12,
> > 	_concurrent/fn-139240b8-8
> >  1cc-4b22-8088-aa5aedd98afe--array//elt-3-13,
> >  _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-14,
> >  _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-15
> > 	partition, PageRank-new.swift, line 38
> > 	shuffle, PageRank-new.swift, line 88
> > 
> > 
> > 
> > However, if I put the stages in Iterate control struct: it works.
> > 
> > //below are main function
> > file input[] <filesys_mapper; location="input", prefix="links-">;
> > file matrix[][];
> > file fn[];
> > 
> > /*iterate loop{
> >    iterate i{
> >       if (i==0){
> >          file score <single_file_mapper; file=@strcat("score.txt.",
> >          @toString(loop))>;
> >          fn = map(input, score);
> >       }
> >       if(i==1){
> >          matrix = shuffle(fn);
> >       }
> >       if(i==2){
> >          file final <single_file_mapper;file=@strcat("score.txt.",
> >          @toString(loop+1))>;
> >          final = reduce(matrix);
> >          file sorted <single_file_mapper;file=@strcat("score.txt.",
> >          @toString(loop+1), ".sorted")>;
> >          sorted = sort(final);
> >       }
> >    }until(i==3);
> > }until(loop==1);*/
> > 
> > 
> > I also checked SwiftMontage implementation, it was also written in
> > this way, so I assumed the first draft should work some time ago.
> > Is
> > this a already known problem?
> > 
> > Best
> > Zhao
> > 
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 


From zhaozhang at uchicago.edu  Sun Jun  2 15:23:29 2013
From: zhaozhang at uchicago.edu (ZHAO ZHANG)
Date: Sun, 2 Jun 2013 13:23:29 -0700
Subject: [Swift-devel] Iterative PageRank in Swift
In-Reply-To: <1428011530.1945766.1370204484519.JavaMail.root@mcs.anl.gov>
References: <1428011530.1945766.1370204484519.JavaMail.root@mcs.anl.gov>
Message-ID: <98C6EACB-9D9B-4FC9-BB9B-1DD9253F6E05@uchicago.edu>

Hi Mike,

Sorry for my late response. I am setting thins up on communicado, and will let you know once it is ready for you to test it.

zhao

On Jun 2, 2013, at 1:21 PM, Michael Wilde wrote:

> Zhao,
> 
> The immediate failure in the run below seems to be due to partition() not creating the following 16 files below its *work* directory:
> 
> _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-0
> _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-1
> ...
> _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-15
> 
> Im assuming mapper.sh gets the name of an element of fn[], which has been mapped by the concurrent mapper, and returns an array mapping of the original name suffixed by -0 through -16?
> 
> Is that the expected behavior?
> 
> Is the partition() app coded to do this, including making the parent directories ./_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array/ ?
> 
> I cant yet explain why/how this worked when you serialized the main (open code) function, but there's enough going on in this code that I'd carefully check the behavior of each stage.
> 
> - Mike
> 
> ----- Original Message -----
>> From: "Michael Wilde" <wilde at mcs.anl.gov>
>> To: "ZHAO ZHANG" <zhaozhang at uchicago.edu>
>> Cc: "swift-devel" <swift-devel at ci.uchicago.edu>
>> Sent: Sunday, June 2, 2013 2:48:58 PM
>> Subject: Re: [Swift-devel] Iterative PageRank in Swift
>> 
>> Zhao, Im studying this.  Can you post a copy of mapper.sh?
>> 
>> Can you put a copy on a local machine here (both failing and working
>> version) that I can experiment with?
>> 
>> Thanks,
>> 
>> - Mike
>> 
>> 
>> ----- Original Message -----
>>> From: "ZHAO ZHANG" <zhaozhang at uchicago.edu>
>>> To: "Swift Devel" <swift-devel at ci.uchicago.edu>
>>> Sent: Sunday, June 2, 2013 2:12:30 PM
>>> Subject: [Swift-devel] Iterative PageRank in Swift
>>> 
>>> Dear all,
>>> 
>>> I have been working with my cousin on an iterative PageRank
>>> implementation with Swift for his graduation project. We now
>>> encounter an problem: we try to use "file fn[]" as intermediate
>>> data
>>> between two stages, however, it does not work well.
>>> 
>>> The app and stage definition looks like below:
>>> zhaozhang at bigben:/var/tmp/workplace$ cat PageRank-new.swift
>>> type file;
>>> 
>>> app (file t) distribution (file f, file s) {
>>>    distribution @filename(f) @filename(t) @filename(s);
>>> }
>>> 
>>> app (file t[]) partition (file f) {
>>>    partition @filename(f) "16";
>>> }
>>> 
>>> app (file t) aggregation (file f[]){
>>>    aggregation @filename(t) @filenames(f);
>>> }
>>> 
>>> app (file t) cat (file f[]){
>>>    cat @filenames(f) stdout=@filename(t);
>>> }
>>> 
>>> app (file t) sort (file f){
>>>    sort "-nrk 2" @filename(f) stdout=@filename(t);
>>> }
>>> 
>>> (file fn[])map(file input[], file score){
>>>   foreach f,i in input {
>>>      file c<regexp_mapper;
>>>         source=@f,
>>>         match="input/(.*)",
>>>         transform="temp/\\1">;
>>>      c = distribution(f, score);
>>>      fn[i] = c;
>>>   }
>>> }
>>> 
>>> (file matrix[][])shuffle(file fn[]){
>>>   foreach c, j in fn{
>>> 	file output[] <ext; exec="bin/mapper.sh", source=@filename(c),
>>> 	scale=16>;
>>> 	output = partition(c);
>>> 	foreach f, k in output{
>>> 		matrix[k][j] = output[k];
>>> 	}
>>>   }
>>> }
>>> 
>>> (file final)reduce(file matrix[][]){
>>>   file result[];
>>>   foreach fl, k in matrix{
>>>      file output <single_file_mapper;
>>>      file=@strcat("result/result-",
>>>      @toString(k))>;
>>>      output = aggregation(fl);
>>>      result[k] = output;
>>>   }
>>> 
>>>   final = cat(result);
>>> }
>>> 
>>> 
>>> If I write the main function as below, it does not work: it seems
>>> the
>>> intermediate files are not mapped to the expected file names.
>>> 
>>> //below are main function
>>> file input[] <filesys_mapper; location="input", prefix="links-">;
>>> file matrix[][];
>>> file fn[];
>>> 
>>> int loop=0;
>>> file score <single_file_mapper; file=@strcat("score.txt.",
>>> @toString(loop))>;
>>> file final <single_file_mapper;file=@strcat("score.txt.",
>>> @toString(loop+1))>;
>>> file sorted <single_file_mapper;file=@strcat("score.txt.",
>>> @toString(loop+1), ".sorted")>;
>>> 
>>> fn = map(input, score);
>>> matrix = shuffle(fn);
>>> final = reduce(matrix);
>>> sorted = sort(final);
>>> 
>>> The execution failed with the following message:
>>> Swift 0.94 swift-r6492 cog-r3658
>>> 
>>> RunID: 20130602-1348-yresjj56
>>> Progress:  time: Sun, 02 Jun 2013 13:48:49 -0500
>>> Progress:  time: Sun, 02 Jun 2013 13:48:51 -0500  Selecting site:3
>>> Checking status:1
>>> Progress:  time: Sun, 02 Jun 2013 13:48:52 -0500  Selecting site:3
>>> Checking status:1  Finished successfully:2
>>> Progress:  time: Sun, 02 Jun 2013 13:48:53 -0500  Selecting site:3
>>> Checking status:1  Finished successfully:4
>>> Execution failed:
>>> 	Exception in partition:
>>>    Arguments: [temp/links-part-0001, 16]
>>>    Host: localhost
>>>    Directory:
>>>    PageRank-new-20130602-1348-yresjj56/jobs/i/partition-ishh5dal
>>>    stderr.txt:
>>>    stdout.txt:
>>> Caused by:
>>> 	The following output files were not created by the application:
>>> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-0,
>>> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-1,
>>> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-2,
>>> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-3,
>>> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-4,
>>> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-5,
>>> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-6,
>>> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-7,
>>> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-8,
>>> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-9,
>>> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-10,
>>> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-11,
>>> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-12,
>>> 	_concurrent/fn-139240b8-8
>>> 1cc-4b22-8088-aa5aedd98afe--array//elt-3-13,
>>> _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-14,
>>> _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-15
>>> 	partition, PageRank-new.swift, line 38
>>> 	shuffle, PageRank-new.swift, line 88
>>> 
>>> 
>>> 
>>> However, if I put the stages in Iterate control struct: it works.
>>> 
>>> //below are main function
>>> file input[] <filesys_mapper; location="input", prefix="links-">;
>>> file matrix[][];
>>> file fn[];
>>> 
>>> /*iterate loop{
>>>   iterate i{
>>>      if (i==0){
>>>         file score <single_file_mapper; file=@strcat("score.txt.",
>>>         @toString(loop))>;
>>>         fn = map(input, score);
>>>      }
>>>      if(i==1){
>>>         matrix = shuffle(fn);
>>>      }
>>>      if(i==2){
>>>         file final <single_file_mapper;file=@strcat("score.txt.",
>>>         @toString(loop+1))>;
>>>         final = reduce(matrix);
>>>         file sorted <single_file_mapper;file=@strcat("score.txt.",
>>>         @toString(loop+1), ".sorted")>;
>>>         sorted = sort(final);
>>>      }
>>>   }until(i==3);
>>> }until(loop==1);*/
>>> 
>>> 
>>> I also checked SwiftMontage implementation, it was also written in
>>> this way, so I assumed the first draft should work some time ago.
>>> Is
>>> this a already known problem?
>>> 
>>> Best
>>> Zhao
>>> 
>>> _______________________________________________
>>> Swift-devel mailing list
>>> Swift-devel at ci.uchicago.edu
>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>> 
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>> 



From wilde at mcs.anl.gov  Sun Jun  2 15:29:04 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Sun, 2 Jun 2013 15:29:04 -0500 (CDT)
Subject: [Swift-devel] Iterative PageRank in Swift
In-Reply-To: <1428011530.1945766.1370204484519.JavaMail.root@mcs.anl.gov>
Message-ID: <472905074.1945832.1370204944524.JavaMail.root@mcs.anl.gov>

I should clarify, that by "Is the partition() app coded to do this, including making the parent directories ./_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array/ ?" I meant:

Is the partition() app coded to *return these exact 16 files*, including making the parent directories ./_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array/ ?

I.e, partition() needs to look at its first command line arg $1 and create $2 (in this case, 16) filenames just like $1, including directories, relative to the current working dir $PWD.

Is it coded to do that?

- Mike

----- Original Message -----
> From: "Michael Wilde" <wilde at mcs.anl.gov>
> To: "ZHAO ZHANG" <zhaozhang at uchicago.edu>
> Cc: "swift-devel" <swift-devel at ci.uchicago.edu>
> Sent: Sunday, June 2, 2013 3:21:24 PM
> Subject: Re: [Swift-devel] Iterative PageRank in Swift
> 
> Zhao,
> 
> The immediate failure in the run below seems to be due to partition()
> not creating the following 16 files below its *work* directory:
> 
> _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-0
> _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-1
> ...
> _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-15
> 
> Im assuming mapper.sh gets the name of an element of fn[], which has
> been mapped by the concurrent mapper, and returns an array mapping
> of the original name suffixed by -0 through -16?
> 
> Is that the expected behavior?
> 
> Is the partition() app coded to do this, including making the parent
> directories
> ./_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array/ ?
> 
> I cant yet explain why/how this worked when you serialized the main
> (open code) function, but there's enough going on in this code that
> I'd carefully check the behavior of each stage.
> 
> - Mike
> 
> ----- Original Message -----
> > From: "Michael Wilde" <wilde at mcs.anl.gov>
> > To: "ZHAO ZHANG" <zhaozhang at uchicago.edu>
> > Cc: "swift-devel" <swift-devel at ci.uchicago.edu>
> > Sent: Sunday, June 2, 2013 2:48:58 PM
> > Subject: Re: [Swift-devel] Iterative PageRank in Swift
> > 
> > Zhao, Im studying this.  Can you post a copy of mapper.sh?
> > 
> > Can you put a copy on a local machine here (both failing and
> > working
> > version) that I can experiment with?
> > 
> > Thanks,
> > 
> > - Mike
> > 
> > 
> > ----- Original Message -----
> > > From: "ZHAO ZHANG" <zhaozhang at uchicago.edu>
> > > To: "Swift Devel" <swift-devel at ci.uchicago.edu>
> > > Sent: Sunday, June 2, 2013 2:12:30 PM
> > > Subject: [Swift-devel] Iterative PageRank in Swift
> > > 
> > > Dear all,
> > > 
> > > I have been working with my cousin on an iterative PageRank
> > > implementation with Swift for his graduation project. We now
> > > encounter an problem: we try to use "file fn[]" as intermediate
> > > data
> > > between two stages, however, it does not work well.
> > > 
> > > The app and stage definition looks like below:
> > > zhaozhang at bigben:/var/tmp/workplace$ cat PageRank-new.swift
> > > type file;
> > > 
> > > app (file t) distribution (file f, file s) {
> > >     distribution @filename(f) @filename(t) @filename(s);
> > > }
> > > 
> > > app (file t[]) partition (file f) {
> > >     partition @filename(f) "16";
> > > }
> > > 
> > > app (file t) aggregation (file f[]){
> > >     aggregation @filename(t) @filenames(f);
> > > }
> > > 
> > > app (file t) cat (file f[]){
> > >     cat @filenames(f) stdout=@filename(t);
> > > }
> > > 
> > > app (file t) sort (file f){
> > >     sort "-nrk 2" @filename(f) stdout=@filename(t);
> > > }
> > > 
> > > (file fn[])map(file input[], file score){
> > >    foreach f,i in input {
> > >       file c<regexp_mapper;
> > >          source=@f,
> > >          match="input/(.*)",
> > >          transform="temp/\\1">;
> > >       c = distribution(f, score);
> > >       fn[i] = c;
> > >    }
> > > }
> > > 
> > > (file matrix[][])shuffle(file fn[]){
> > >    foreach c, j in fn{
> > > 	file output[] <ext; exec="bin/mapper.sh", source=@filename(c),
> > > 	scale=16>;
> > > 	output = partition(c);
> > > 	foreach f, k in output{
> > > 		matrix[k][j] = output[k];
> > > 	}
> > >    }
> > > }
> > > 
> > > (file final)reduce(file matrix[][]){
> > >    file result[];
> > >    foreach fl, k in matrix{
> > >       file output <single_file_mapper;
> > >       file=@strcat("result/result-",
> > >       @toString(k))>;
> > >       output = aggregation(fl);
> > >       result[k] = output;
> > >    }
> > > 
> > >    final = cat(result);
> > > }
> > > 
> > > 
> > > If I write the main function as below, it does not work: it seems
> > > the
> > > intermediate files are not mapped to the expected file names.
> > > 
> > > //below are main function
> > > file input[] <filesys_mapper; location="input", prefix="links-">;
> > > file matrix[][];
> > > file fn[];
> > > 
> > > int loop=0;
> > > file score <single_file_mapper; file=@strcat("score.txt.",
> > > @toString(loop))>;
> > > file final <single_file_mapper;file=@strcat("score.txt.",
> > > @toString(loop+1))>;
> > > file sorted <single_file_mapper;file=@strcat("score.txt.",
> > > @toString(loop+1), ".sorted")>;
> > > 
> > > fn = map(input, score);
> > > matrix = shuffle(fn);
> > > final = reduce(matrix);
> > > sorted = sort(final);
> > > 
> > > The execution failed with the following message:
> > > Swift 0.94 swift-r6492 cog-r3658
> > > 
> > > RunID: 20130602-1348-yresjj56
> > > Progress:  time: Sun, 02 Jun 2013 13:48:49 -0500
> > > Progress:  time: Sun, 02 Jun 2013 13:48:51 -0500  Selecting
> > > site:3
> > >  Checking status:1
> > > Progress:  time: Sun, 02 Jun 2013 13:48:52 -0500  Selecting
> > > site:3
> > >  Checking status:1  Finished successfully:2
> > > Progress:  time: Sun, 02 Jun 2013 13:48:53 -0500  Selecting
> > > site:3
> > >  Checking status:1  Finished successfully:4
> > > Execution failed:
> > > 	Exception in partition:
> > >     Arguments: [temp/links-part-0001, 16]
> > >     Host: localhost
> > >     Directory:
> > >     PageRank-new-20130602-1348-yresjj56/jobs/i/partition-ishh5dal
> > >     stderr.txt:
> > >     stdout.txt:
> > > Caused by:
> > > 	The following output files were not created by the application:
> > > 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-0,
> > > 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-1,
> > > 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-2,
> > > 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-3,
> > > 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-4,
> > > 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-5,
> > > 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-6,
> > > 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-7,
> > > 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-8,
> > > 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-9,
> > > 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-10,
> > > 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-11,
> > > 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-12,
> > > 	_concurrent/fn-139240b8-8
> > >  1cc-4b22-8088-aa5aedd98afe--array//elt-3-13,
> > >  _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-14,
> > >  _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-15
> > > 	partition, PageRank-new.swift, line 38
> > > 	shuffle, PageRank-new.swift, line 88
> > > 
> > > 
> > > 
> > > However, if I put the stages in Iterate control struct: it works.
> > > 
> > > //below are main function
> > > file input[] <filesys_mapper; location="input", prefix="links-">;
> > > file matrix[][];
> > > file fn[];
> > > 
> > > /*iterate loop{
> > >    iterate i{
> > >       if (i==0){
> > >          file score <single_file_mapper;
> > >          file=@strcat("score.txt.",
> > >          @toString(loop))>;
> > >          fn = map(input, score);
> > >       }
> > >       if(i==1){
> > >          matrix = shuffle(fn);
> > >       }
> > >       if(i==2){
> > >          file final
> > >          <single_file_mapper;file=@strcat("score.txt.",
> > >          @toString(loop+1))>;
> > >          final = reduce(matrix);
> > >          file sorted
> > >          <single_file_mapper;file=@strcat("score.txt.",
> > >          @toString(loop+1), ".sorted")>;
> > >          sorted = sort(final);
> > >       }
> > >    }until(i==3);
> > > }until(loop==1);*/
> > > 
> > > 
> > > I also checked SwiftMontage implementation, it was also written
> > > in
> > > this way, so I assumed the first draft should work some time ago.
> > > Is
> > > this a already known problem?
> > > 
> > > Best
> > > Zhao
> > > 
> > > _______________________________________________
> > > Swift-devel mailing list
> > > Swift-devel at ci.uchicago.edu
> > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > 
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 


From zhaozhang at uchicago.edu  Sun Jun  2 15:41:14 2013
From: zhaozhang at uchicago.edu (ZHAO ZHANG)
Date: Sun, 2 Jun 2013 13:41:14 -0700
Subject: [Swift-devel] Iterative PageRank in Swift
In-Reply-To: <472905074.1945832.1370204944524.JavaMail.root@mcs.anl.gov>
References: <472905074.1945832.1370204944524.JavaMail.root@mcs.anl.gov>
Message-ID: <A1048A9F-7907-4222-947B-D83914E08380@uchicago.edu>

Hi Mike,

On Jun 2, 2013, at 1:29 PM, Michael Wilde wrote:

> I should clarify, that by "Is the partition() app coded to do this, including making the parent directories ./_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array/ ?" I meant:
> 
> Is the partition() app coded to *return these exact 16 files*, including making the parent directories ./_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array/ ?

Ah, I see what you mean here. In my partition app definition, 
app (file t[]) partition (file f) {
    partition @filename(f) "16";
}

I did not put output files as parameter. I think that is why it does not work right.
The command executes as:
bin/partition.py input_file 16 

and it produces 16 output files with the naming convention of input_file-0, input-file-1, ..., input-file-15

Now I think of one way to correct this.

I will be offline for the next couple of hours. I will let you know my progress once I figured it out. You don't have to try anything for now, as I think I know there the problem is.

best
Zhao

> 
> I.e, partition() needs to look at its first command line arg $1 and create $2 (in this case, 16) filenames just like $1, including directories, relative to the current working dir $PWD.
> 
> Is it coded to do that?
> 
> - Mike
> 
> ----- Original Message -----
>> From: "Michael Wilde" <wilde at mcs.anl.gov>
>> To: "ZHAO ZHANG" <zhaozhang at uchicago.edu>
>> Cc: "swift-devel" <swift-devel at ci.uchicago.edu>
>> Sent: Sunday, June 2, 2013 3:21:24 PM
>> Subject: Re: [Swift-devel] Iterative PageRank in Swift
>> 
>> Zhao,
>> 
>> The immediate failure in the run below seems to be due to partition()
>> not creating the following 16 files below its *work* directory:
>> 
>> _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-0
>> _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-1
>> ...
>> _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-15
>> 
>> Im assuming mapper.sh gets the name of an element of fn[], which has
>> been mapped by the concurrent mapper, and returns an array mapping
>> of the original name suffixed by -0 through -16?
>> 
>> Is that the expected behavior?
>> 
>> Is the partition() app coded to do this, including making the parent
>> directories
>> ./_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array/ ?
>> 
>> I cant yet explain why/how this worked when you serialized the main
>> (open code) function, but there's enough going on in this code that
>> I'd carefully check the behavior of each stage.
>> 
>> - Mike
>> 
>> ----- Original Message -----
>>> From: "Michael Wilde" <wilde at mcs.anl.gov>
>>> To: "ZHAO ZHANG" <zhaozhang at uchicago.edu>
>>> Cc: "swift-devel" <swift-devel at ci.uchicago.edu>
>>> Sent: Sunday, June 2, 2013 2:48:58 PM
>>> Subject: Re: [Swift-devel] Iterative PageRank in Swift
>>> 
>>> Zhao, Im studying this.  Can you post a copy of mapper.sh?
>>> 
>>> Can you put a copy on a local machine here (both failing and
>>> working
>>> version) that I can experiment with?
>>> 
>>> Thanks,
>>> 
>>> - Mike
>>> 
>>> 
>>> ----- Original Message -----
>>>> From: "ZHAO ZHANG" <zhaozhang at uchicago.edu>
>>>> To: "Swift Devel" <swift-devel at ci.uchicago.edu>
>>>> Sent: Sunday, June 2, 2013 2:12:30 PM
>>>> Subject: [Swift-devel] Iterative PageRank in Swift
>>>> 
>>>> Dear all,
>>>> 
>>>> I have been working with my cousin on an iterative PageRank
>>>> implementation with Swift for his graduation project. We now
>>>> encounter an problem: we try to use "file fn[]" as intermediate
>>>> data
>>>> between two stages, however, it does not work well.
>>>> 
>>>> The app and stage definition looks like below:
>>>> zhaozhang at bigben:/var/tmp/workplace$ cat PageRank-new.swift
>>>> type file;
>>>> 
>>>> app (file t) distribution (file f, file s) {
>>>>    distribution @filename(f) @filename(t) @filename(s);
>>>> }
>>>> 
>>>> app (file t[]) partition (file f) {
>>>>    partition @filename(f) "16";
>>>> }
>>>> 
>>>> app (file t) aggregation (file f[]){
>>>>    aggregation @filename(t) @filenames(f);
>>>> }
>>>> 
>>>> app (file t) cat (file f[]){
>>>>    cat @filenames(f) stdout=@filename(t);
>>>> }
>>>> 
>>>> app (file t) sort (file f){
>>>>    sort "-nrk 2" @filename(f) stdout=@filename(t);
>>>> }
>>>> 
>>>> (file fn[])map(file input[], file score){
>>>>   foreach f,i in input {
>>>>      file c<regexp_mapper;
>>>>         source=@f,
>>>>         match="input/(.*)",
>>>>         transform="temp/\\1">;
>>>>      c = distribution(f, score);
>>>>      fn[i] = c;
>>>>   }
>>>> }
>>>> 
>>>> (file matrix[][])shuffle(file fn[]){
>>>>   foreach c, j in fn{
>>>> 	file output[] <ext; exec="bin/mapper.sh", source=@filename(c),
>>>> 	scale=16>;
>>>> 	output = partition(c);
>>>> 	foreach f, k in output{
>>>> 		matrix[k][j] = output[k];
>>>> 	}
>>>>   }
>>>> }
>>>> 
>>>> (file final)reduce(file matrix[][]){
>>>>   file result[];
>>>>   foreach fl, k in matrix{
>>>>      file output <single_file_mapper;
>>>>      file=@strcat("result/result-",
>>>>      @toString(k))>;
>>>>      output = aggregation(fl);
>>>>      result[k] = output;
>>>>   }
>>>> 
>>>>   final = cat(result);
>>>> }
>>>> 
>>>> 
>>>> If I write the main function as below, it does not work: it seems
>>>> the
>>>> intermediate files are not mapped to the expected file names.
>>>> 
>>>> //below are main function
>>>> file input[] <filesys_mapper; location="input", prefix="links-">;
>>>> file matrix[][];
>>>> file fn[];
>>>> 
>>>> int loop=0;
>>>> file score <single_file_mapper; file=@strcat("score.txt.",
>>>> @toString(loop))>;
>>>> file final <single_file_mapper;file=@strcat("score.txt.",
>>>> @toString(loop+1))>;
>>>> file sorted <single_file_mapper;file=@strcat("score.txt.",
>>>> @toString(loop+1), ".sorted")>;
>>>> 
>>>> fn = map(input, score);
>>>> matrix = shuffle(fn);
>>>> final = reduce(matrix);
>>>> sorted = sort(final);
>>>> 
>>>> The execution failed with the following message:
>>>> Swift 0.94 swift-r6492 cog-r3658
>>>> 
>>>> RunID: 20130602-1348-yresjj56
>>>> Progress:  time: Sun, 02 Jun 2013 13:48:49 -0500
>>>> Progress:  time: Sun, 02 Jun 2013 13:48:51 -0500  Selecting
>>>> site:3
>>>> Checking status:1
>>>> Progress:  time: Sun, 02 Jun 2013 13:48:52 -0500  Selecting
>>>> site:3
>>>> Checking status:1  Finished successfully:2
>>>> Progress:  time: Sun, 02 Jun 2013 13:48:53 -0500  Selecting
>>>> site:3
>>>> Checking status:1  Finished successfully:4
>>>> Execution failed:
>>>> 	Exception in partition:
>>>>    Arguments: [temp/links-part-0001, 16]
>>>>    Host: localhost
>>>>    Directory:
>>>>    PageRank-new-20130602-1348-yresjj56/jobs/i/partition-ishh5dal
>>>>    stderr.txt:
>>>>    stdout.txt:
>>>> Caused by:
>>>> 	The following output files were not created by the application:
>>>> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-0,
>>>> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-1,
>>>> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-2,
>>>> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-3,
>>>> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-4,
>>>> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-5,
>>>> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-6,
>>>> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-7,
>>>> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-8,
>>>> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-9,
>>>> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-10,
>>>> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-11,
>>>> 	_concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-12,
>>>> 	_concurrent/fn-139240b8-8
>>>> 1cc-4b22-8088-aa5aedd98afe--array//elt-3-13,
>>>> _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-14,
>>>> _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-15
>>>> 	partition, PageRank-new.swift, line 38
>>>> 	shuffle, PageRank-new.swift, line 88
>>>> 
>>>> 
>>>> 
>>>> However, if I put the stages in Iterate control struct: it works.
>>>> 
>>>> //below are main function
>>>> file input[] <filesys_mapper; location="input", prefix="links-">;
>>>> file matrix[][];
>>>> file fn[];
>>>> 
>>>> /*iterate loop{
>>>>   iterate i{
>>>>      if (i==0){
>>>>         file score <single_file_mapper;
>>>>         file=@strcat("score.txt.",
>>>>         @toString(loop))>;
>>>>         fn = map(input, score);
>>>>      }
>>>>      if(i==1){
>>>>         matrix = shuffle(fn);
>>>>      }
>>>>      if(i==2){
>>>>         file final
>>>>         <single_file_mapper;file=@strcat("score.txt.",
>>>>         @toString(loop+1))>;
>>>>         final = reduce(matrix);
>>>>         file sorted
>>>>         <single_file_mapper;file=@strcat("score.txt.",
>>>>         @toString(loop+1), ".sorted")>;
>>>>         sorted = sort(final);
>>>>      }
>>>>   }until(i==3);
>>>> }until(loop==1);*/
>>>> 
>>>> 
>>>> I also checked SwiftMontage implementation, it was also written
>>>> in
>>>> this way, so I assumed the first draft should work some time ago.
>>>> Is
>>>> this a already known problem?
>>>> 
>>>> Best
>>>> Zhao
>>>> 
>>>> _______________________________________________
>>>> Swift-devel mailing list
>>>> Swift-devel at ci.uchicago.edu
>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>>> 
>>> _______________________________________________
>>> Swift-devel mailing list
>>> Swift-devel at ci.uchicago.edu
>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>> 
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>> 



From wilde at mcs.anl.gov  Sun Jun  2 15:50:40 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Sun, 2 Jun 2013 15:50:40 -0500 (CDT)
Subject: [Swift-devel] Iterative PageRank in Swift
In-Reply-To: <A1048A9F-7907-4222-947B-D83914E08380@uchicago.edu>
Message-ID: <1542136690.1946178.1370206240949.JavaMail.root@mcs.anl.gov>


> ... In my partition app definition,
> app (file t[]) partition (file f) {
>     partition @filename(f) "16";
> }
> 
> I did not put output files as parameter. I think that is why it does
> not work right.
> The command executes as:
> bin/partition.py input_file 16
> 
> and it produces 16 output files with the naming convention of
> input_file-0, input-file-1, ..., input-file-15

That seems close to OK as-is.  The output files are OK as a parameter to the swift app function partition(). This will cause Swift to look for that array of files and pass them to any downstream functions in the data flow that depend on them.  They don't need to be mentioned on the partition *command line* so long as partition.py correctly creates these 16 files.

The first problem I would look for here is whether partition will do so correctly if the input path name has directory components in it.  In other words, given this command line:

  bin/partition.py d1/d2/input-file 16

it should create ./d1/d2/input-file-0 ... ./d1/d2/input-file-15, NOT ./input-file-16

You can check if its doing this both standalone, and under Swift by doing a find on the job dir in the workdirectory.

- Mike


From wilde at mcs.anl.gov  Sun Jun  2 16:43:42 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Sun, 2 Jun 2013 16:43:42 -0500 (CDT)
Subject: [Swift-devel] Is bugzilla search broken?
Message-ID: <817781529.1946837.1370209422036.JavaMail.root@mcs.anl.gov>

Im getting web server 500 error from Swift bugzilla when I try to do any kind of search.

Is anyone else seeing the same problem?

Thanks,

- Mike


From wilde at mcs.anl.gov  Sun Jun  2 16:45:53 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Sun, 2 Jun 2013 16:45:53 -0500 (CDT)
Subject: [Swift-devel] Is bugzilla search broken?
In-Reply-To: <817781529.1946837.1370209422036.JavaMail.root@mcs.anl.gov>
Message-ID: <1424558025.1946863.1370209553836.JavaMail.root@mcs.anl.gov>

Sorry, not clear its a "500".  Just by typing a string in the top search box and clicking "Find", as well as from any detailed search, I get this:

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, webmaster at mcs.anl.gov and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

- Mike


----- Original Message -----
> From: "Michael Wilde" <wilde at mcs.anl.gov>
> To: "swift-devel" <swift-devel at ci.uchicago.edu>
> Sent: Sunday, June 2, 2013 4:43:42 PM
> Subject: [Swift-devel] Is bugzilla search broken?
> 
> Im getting web server 500 error from Swift bugzilla when I try to do
> any kind of search.
> 
> Is anyone else seeing the same problem?
> 
> Thanks,
> 
> - Mike
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 


From hategan at mcs.anl.gov  Sun Jun  2 17:17:39 2013
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 02 Jun 2013 15:17:39 -0700
Subject: [Swift-devel] Is bugzilla search broken?
In-Reply-To: <1424558025.1946863.1370209553836.JavaMail.root@mcs.anl.gov>
References: <1424558025.1946863.1370209553836.JavaMail.root@mcs.anl.gov>
Message-ID: <1370211459.10963.0.camel@echo>

I'm getting that too. We should email systems.

Mihael

On Sun, 2013-06-02 at 16:45 -0500, Michael Wilde wrote:
> Sorry, not clear its a "500".  Just by typing a string in the top search box and clicking "Find", as well as from any detailed search, I get this:
> 
> Internal Server Error
> 
> The server encountered an internal error or misconfiguration and was unable to complete your request.
> 
> Please contact the server administrator, webmaster at mcs.anl.gov and inform them of the time the error occurred, and anything you might have done that may have caused the error.
> 
> More information about this error may be available in the server error log.
> 
> - Mike
> 
> 
> ----- Original Message -----
> > From: "Michael Wilde" <wilde at mcs.anl.gov>
> > To: "swift-devel" <swift-devel at ci.uchicago.edu>
> > Sent: Sunday, June 2, 2013 4:43:42 PM
> > Subject: [Swift-devel] Is bugzilla search broken?
> > 
> > Im getting web server 500 error from Swift bugzilla when I try to do
> > any kind of search.
> > 
> > Is anyone else seeing the same problem?
> > 
> > Thanks,
> > 
> > - Mike
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel




From wilde at mcs.anl.gov  Sun Jun  2 17:22:28 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Sun, 2 Jun 2013 17:22:28 -0500 (CDT)
Subject: [Swift-devel] Is bugzilla search broken?
In-Reply-To: <1370211459.10963.0.camel@echo>
Message-ID: <499674510.1948509.1370211748895.JavaMail.root@mcs.anl.gov>


> I'm getting that too. We should email systems.

Thanks.  I did.


From zhaozhang at uchicago.edu  Sun Jun  2 20:12:24 2013
From: zhaozhang at uchicago.edu (ZHAO ZHANG)
Date: Sun, 2 Jun 2013 18:12:24 -0700
Subject: [Swift-devel] Iterative PageRank in Swift
In-Reply-To: <6F85720E-81C4-42BD-B12F-6DF281DCE023@uchicago.edu>
References: <6F85720E-81C4-42BD-B12F-6DF281DCE023@uchicago.edu>
Message-ID: <AEAB9507-3930-4F8A-8271-C03C4F9F825B@uchicago.edu>

Thanks Mike. The problem has been solved. The reason is that the output files of partition app were not properly mapped.

best
zhao

On Jun 2, 2013, at 12:12 PM, ZHAO ZHANG wrote:

> Dear all,
> 
> I have been working with my cousin on an iterative PageRank implementation with Swift for his graduation project. We now encounter an problem: we try to use "file fn[]" as intermediate data between two stages, however, it does not work well. 
> 
> The app and stage definition looks like below:
> zhaozhang at bigben:/var/tmp/workplace$ cat PageRank-new.swift
> type file; 
> 
> app (file t) distribution (file f, file s) {   
>    distribution @filename(f) @filename(t) @filename(s);
> }
> 
> app (file t[]) partition (file f) {   
>    partition @filename(f) "16";
> }
> 
> app (file t) aggregation (file f[]){
>    aggregation @filename(t) @filenames(f);
> }
> 
> app (file t) cat (file f[]){
>    cat @filenames(f) stdout=@filename(t);
> }
> 
> app (file t) sort (file f){
>    sort "-nrk 2" @filename(f) stdout=@filename(t);
> }
> 
> (file fn[])map(file input[], file score){
>   foreach f,i in input {
>      file c<regexp_mapper;
>         source=@f,
>         match="input/(.*)",
>         transform="temp/\\1">;
>      c = distribution(f, score);  
>      fn[i] = c;
>   }
> }
> 
> (file matrix[][])shuffle(file fn[]){
>   foreach c, j in fn{
> 	file output[] <ext; exec="bin/mapper.sh", source=@filename(c), scale=16>;
> 	output = partition(c);
> 	foreach f, k in output{
> 		matrix[k][j] = output[k];
> 	}
>   }
> }
> 
> (file final)reduce(file matrix[][]){
>   file result[];	  
>   foreach fl, k in matrix{
>      file output <single_file_mapper; file=@strcat("result/result-", @toString(k))>;
>      output = aggregation(fl);
>      result[k] = output;
>   }   
> 
>   final = cat(result);
> }
> 
> 
> If I write the main function as below, it does not work: it seems the intermediate files are not mapped to the expected file names.
> 
> //below are main function
> file input[] <filesys_mapper; location="input", prefix="links-">;
> file matrix[][];
> file fn[];
> 
> int loop=0;
> file score <single_file_mapper; file=@strcat("score.txt.", @toString(loop))>;
> file final <single_file_mapper;file=@strcat("score.txt.", @toString(loop+1))>;
> file sorted <single_file_mapper;file=@strcat("score.txt.", @toString(loop+1), ".sorted")>;
> 
> fn = map(input, score);
> matrix = shuffle(fn);
> final = reduce(matrix);
> sorted = sort(final);
> 
> The execution failed with the following message:
> Swift 0.94 swift-r6492 cog-r3658
> 
> RunID: 20130602-1348-yresjj56
> Progress:  time: Sun, 02 Jun 2013 13:48:49 -0500
> Progress:  time: Sun, 02 Jun 2013 13:48:51 -0500  Selecting site:3  Checking status:1
> Progress:  time: Sun, 02 Jun 2013 13:48:52 -0500  Selecting site:3  Checking status:1  Finished successfully:2
> Progress:  time: Sun, 02 Jun 2013 13:48:53 -0500  Selecting site:3  Checking status:1  Finished successfully:4
> Execution failed:
> 	Exception in partition:
>    Arguments: [temp/links-part-0001, 16]
>    Host: localhost
>    Directory: PageRank-new-20130602-1348-yresjj56/jobs/i/partition-ishh5dal
>    stderr.txt: 
>    stdout.txt: 
> Caused by:
> 	The following output files were not created by the application: _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-0, _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-1, _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-2, _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-3, _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-4, _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-5, _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-6, _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-7, _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-8, _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-9, _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-10, _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-11, _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-12, _concurrent/fn-139240b8-8
> 1cc-4b22-8088-aa5aedd98afe--array//elt-3-13, _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-14, _concurrent/fn-139240b8-81cc-4b22-8088-aa5aedd98afe--array//elt-3-15
> 	partition, PageRank-new.swift, line 38
> 	shuffle, PageRank-new.swift, line 88
> 
> 
> 
> However, if I put the stages in Iterate control struct: it works.
> 
> //below are main function
> file input[] <filesys_mapper; location="input", prefix="links-">;
> file matrix[][];
> file fn[];
> 
> /*iterate loop{
>   iterate i{
>      if (i==0){
>         file score <single_file_mapper; file=@strcat("score.txt.", @toString(loop))>;
>         fn = map(input, score);
>      }
>      if(i==1){
>         matrix = shuffle(fn);
>      }
>      if(i==2){
>         file final <single_file_mapper;file=@strcat("score.txt.", @toString(loop+1))>;
>         final = reduce(matrix);
>         file sorted <single_file_mapper;file=@strcat("score.txt.", @toString(loop+1), ".sorted")>;
>         sorted = sort(final);
>      }
>   }until(i==3);
> }until(loop==1);*/
> 
> 
> I also checked SwiftMontage implementation, it was also written in this way, so I assumed the first draft should work some time ago. Is this a already known problem?
> 
> Best
> Zhao
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel



From hategan at mcs.anl.gov  Sun Jun  2 21:44:09 2013
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 02 Jun 2013 19:44:09 -0700
Subject: [Swift-devel] sourceforge SVN changes
Message-ID: <1370227449.30798.4.camel@echo>

Hi,

Sourceforge has migrated the svn repos. If you have an existing checkout
you should:

1. make sure you have your public key in your account:
https://sourceforge.net/account/ssh

2. do a 'svn relocate "svn
+ssh://<username>@svn.code.sf.net/p/cogkit/svn/"'

Also, we should update our checkout instructions. I believe they still
support https in theory, but when I tried "https" instead of "svn+ssh" I
got an "Internal server error".

Mihael



From wilde at mcs.anl.gov  Tue Jun  4 11:56:52 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Tue, 4 Jun 2013 11:56:52 -0500 (CDT)
Subject: [Swift-devel] SWIFT jobs in C/X states on uc3-sub
In-Reply-To: <A7541FF2-776F-4830-ADF4-3180793ED658@uchicago.edu>
Message-ID: <110886739.113278.1370365012984.JavaMail.root@mcs.anl.gov>

Hi Lincoln,

No update yet.  I just filed this as Swift bug 1010 assigned it to you, David.

Lincoln, can you help us on this?  Below is a sample of the Condor submit script generated by Swift.

Is our problem simply caused by the "leave_in_queue" flag being TRUE?  I think a few simple experiments with the .submit file below should help re-create and fix the problem.

Also, I see that our stdout and stderr files are empty.  David, should be be (optionally) capturing a per-job condor .log file?

Lincoln, any other guidance on the contents of the submit file?

- Mike

should_transfer_files = YES
when_to_transfer_output = ON_EXIT_OR_EVICT
Transfer_Executable = false
machine_count = 1
output = /home/wilde/.globus/scripts/Condor2152159861288524507.submit.stdout
error = /home/wilde/.globus/scripts/Condor2152159861288524507.submit.stderr
environment = WORKER_LOGGING_LEVEL=NONE;
executable = /usr/bin/perl
arguments = cscript5775404700699952879.pl http://10.1.3.94:53610,http://128.135.158.243:53610 0523-5710180-000010 NOLOGGING
transfer_input_files = /home/wilde/.globus/coasters/cscript5775404700699952879.pl
requirements = regexp("uc3-c*", Machine)
+accountinggroup = "group_friends.wilde"
notification = Never
leave_in_queue = TRUE
queue


----- Original Message -----
> From: "Lincoln Bryant" <lincolnb at uchicago.edu>
> To: "Michael Wilde" <wilde at mcs.anl.gov>
> Cc: "David Kelly" <davidk at ci.uchicago.edu>, "swift-devel" <swift-devel at ci.uchicago.edu>, "uc3-support"
> <uc3-support at lists.uchicago.edu>
> Sent: Tuesday, June 4, 2013 11:34:42 AM
> Subject: Re: SWIFT jobs in C/X states on uc3-sub
> 
> Hi Mike,
> 
> I was just wondering if there was any update on this. I still see
> Swift jobs sit in the C state in our Condor pool after they've
> finished.
> 
> > [lincolnb at uc3-sub local]$ condor_q
> > 
> > 
> > -- Submitter: uc3-sub.uchicago.edu :
> > <10.1.3.94:9618?sock=25212_0c25_38> : uc3-sub.uchicago.edu
> >  ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD
> > 110372.0   maheshwari      6/4  09:19   0+00:21:40 C  0   732.4
> > perl cscript948689
> > 110373.0   maheshwari      6/4  09:19   0+00:21:39 C  0   732.4
> > perl cscript948689
> > 110374.0   maheshwari      6/4  09:19   0+00:21:40 C  0   732.4
> > perl cscript948689
> > 110375.0   maheshwari      6/4  09:19   0+00:21:39 C  0   732.4
> > perl cscript948689
> > 110376.0   maheshwari      6/4  09:19   0+00:21:39 C  0   732.4
> > perl cscript948689
> > 110377.0   maheshwari      6/4  09:19   0+00:21:39 C  0   732.4
> > perl cscript948689
> > 110378.0   maheshwari      6/4  09:19   0+00:21:39 C  0   732.4
> > perl cscript948689
> > 110379.0   maheshwari      6/4  09:19   0+00:21:41 C  0   463.9
> > perl cscript948689
> 
> 
> I have swift-0.94 installed on uc3-sub.uchicago.edu
> 
> Cheers,
> Lincoln Bryant
> 
> On Mar 20, 2013, at 11:58 AM, Michael Wilde wrote:
> 
> > My jobs must be fossils; David, Yadu, we should test whether and
> > why Swift doesnt always clean up.
> > 
> > I realize that if Swift hangs and needs to be SIGKILL'ed then it
> > cant.  But lets see if the Condor provider is cleaning up when
> > Swift gets a catchable signal.
> > 
> > Lincoln, lease remove the "wilde" jobs if you can do that.
> > 
> > Thanks,
> > 
> > - Mike
> > 
> > ----- Original Message -----
> >> From: "Lincoln Bryant" <lincolnb at uchicago.edu>
> >> To: "David Kelly" <davidk at ci.uchicago.edu>
> >> Cc: "Michael Wilde" <wilde at mcs.anl.gov>
> >> Sent: Wednesday, March 20, 2013 11:26:30 AM
> >> Subject: SWIFT jobs in C/X states on uc3-sub
> >> 
> >> Hi David / Mike,
> >> 
> >> I notice there are a lot of old jobs sitting in the UC3 queue.
> >> They're sitting in either "X" (removed) or "C" (completed). Sample
> >> below:
> >> 
> >>> 70980.0   wilde           3/12 11:42   0+00:18:59 C  0   43.9
> >>> perl
> >>> cscript906755
> >>> 70981.0   wilde           3/12 11:42   0+00:03:14 C  0   46.4
> >>> perl
> >>> cscript906755
> >>> 70982.0   wilde           3/12 11:42   0+00:03:14 C  0   46.4
> >>> perl
> >>> cscript906755
> >> 
> >>> 71652.0   davidk          3/13 19:28   0+00:00:01 X  0   0.0
> >>>  perl
> >>> cscript500002
> >>> 71653.0   davidk          3/13 19:28   0+00:00:01 X  0   0.0
> >>>  perl
> >>> cscript500002
> >>> 71896.0   davidk          3/13 19:38   0+00:00:01 X  0   0.0
> >>>  perl
> >>> cscript339551
> >> 
> >> Are these jobs completing OK on your side?
> >> 
> >> Occasionally I go in and purge old jobs, but I'm curious if
> >> there's
> >> something in your submit files that is causing them to stick.
> >> 
> >> Cheers,
> >> Lincoln
> 
> 


From yadudoc1729 at gmail.com  Tue Jun  4 16:29:55 2013
From: yadudoc1729 at gmail.com (Yadu Nand)
Date: Tue, 4 Jun 2013 16:29:55 -0500
Subject: [Swift-devel] A bunch of glitches
Message-ID: <CANa904n9X4zqtThOSCo_=FdG5z4kbRW17i6gR+XxMXC7sD2AxA@mail.gmail.com>

Hi Everyone,

I've been putting down a list of potential pitfalls and swift quirks,
which I think could help a newbie.
The content is not formatted or cleaned up yet, but the document is
public and searchable, so the
error strings should easily bring you to the relevant area.

Please feel free to edit or add content.

https://docs.google.com/document/d/1KBqryISlxV2X3eo12u5ilhR6Vm_pJNE8C-8KGAjusOE/edit?usp=sharing

-Yadu


From davidk at ci.uchicago.edu  Wed Jun  5 13:07:46 2013
From: davidk at ci.uchicago.edu (David Kelly)
Date: Wed, 5 Jun 2013 13:07:46 -0500 (CDT)
Subject: [Swift-devel] Domain name
In-Reply-To: <1192744849.2045978.1369441527231.JavaMail.root@mcs.anl.gov>
Message-ID: <1754623550.2636252.1370455666048.JavaMail.root@ci.uchicago.edu>

Hello all, 


Mike wanted me to email the list to see if there was a consensus on what our primary domain name should be. I think the idea is to have one domain name be the official site, then have the others redirect there. The list we have: 



swiftlang.org 
swiftlang.com 
swiftlang.net 
swift-lang.org 
swift-lang.com 
swift-lang.net 


At the moment, swiftlang.org and swift-lang.org are both configured to load the Swift page. 


My preference is swift-lang.org. 


David 


----- Original Message -----


From: "Michael Wilde" <wilde at mcs.anl.gov> 
To: "swift-devel" <swift-devel at ci.uchicago.edu> 
Sent: Friday, May 24, 2013 7:25:27 PM 
Subject: Re: [Swift-devel] Domain name 

I got swiftlang.org and swift-lang.org 

We can initially forward them to the www.ci.uchicago.edu/swift site, then later get the site to appear as if its hosted at the new domain name. 

- Mike 

----- Original Message ----- 
> From: "Tim Armstrong" <tim.g.armstrong at gmail.com> 
> To: "swift-devel" <swift-devel at ci.uchicago.edu> 
> Sent: Friday, May 24, 2013 5:01:33 PM 
> Subject: [Swift-devel] Domain name 
> 
> 
> 
> 
> Following up on the earlier discussion, swift-lang.org isn't taken. 
> 
> 
> - Tim 
> 
> _______________________________________________ 
> Swift-devel mailing list 
> Swift-devel at ci.uchicago.edu 
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel 
> 
_______________________________________________ 
Swift-devel mailing list 
Swift-devel at ci.uchicago.edu 
https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel 



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130605/893d68c7/attachment.html>

From wilde at mcs.anl.gov  Wed Jun  5 13:39:37 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 5 Jun 2013 13:39:37 -0500 (CDT)
Subject: [Swift-devel] Domain name
In-Reply-To: <etPan.51af80ba.4e6afb66.c4@Vytas-Surrogate.local>
Message-ID: <1529864766.497163.1370457577538.JavaMail.root@mcs.anl.gov>

We're trying to deprecate the use of the name "SwiftScript". Its a bit hard to pronounce (rolls off the tongue poorly) and we want to be more selective in whether we pitch it as a scripting language or parallel programming language.

We need to select one URL like http://swift-lang.org and all other URLs use should bring you to the site under this preferred URL.

I agree with David on preferring swift-lang.org: it "looks" best, even if we occasionally have to point out the hyphen.

Vytas: any advice on how best to do this and what to watch out for? Is it simply a matter of setting the site to redirect all entry points to one primary URL?

- Mike

----- Original Message -----
> From: "Vytas Cuplinskas" <vytas at ci.uchicago.edu>
> To: "David Kelly" <davidk at ci.uchicago.edu>, "swift-devel" <swift-devel at ci.uchicago.edu>
> Cc: "Michael Wilde" <wilde at mcs.anl.gov>
> Sent: Wednesday, June 5, 2013 1:17:30 PM
> Subject: Re: [Swift-devel] Domain name
> 
> 
> 
> Just curious ? did you ever considered swiftscript.org?
> 
> 
> Vytas
> 
> 
> 
> P.S. if this is opening a Pandora's box please disregard
> 
> 
> 
> On June 5, 2013 at 1:07:46 PM, David Kelly (davidk at ci.uchicago.edu)
> wrote:
> 
> 
> 
> Hello all,
> 
> 
> Mike wanted me to email the list to see if there was a consensus on
> what our primary domain name should be. I think the idea is to have
> one domain name be the official site, then have the others redirect
> there. The list we have:
> 
> 
> 
> swiftlang.org
> swiftlang.com
> swiftlang.net
> swift-lang.org
> swift-lang.com
> swift-lang.net
> 
> 
> At the moment, swiftlang.org and swift-lang.org are both configured
> to load the Swift page.
> 
> 
> My preference is swift-lang.org.
> 
> 
> David
> 
> 
> ----- Original Message -----
> 
> 
> From: "Michael Wilde" <wilde at mcs.anl.gov>
> To: "swift-devel" <swift-devel at ci.uchicago.edu>
> Sent: Friday, May 24, 2013 7:25:27 PM
> Subject: Re: [Swift-devel] Domain name
> 
> I got swiftlang.org and swift-lang.org
> 
> We can initially forward them to the www.ci.uchicago.edu/swift site,
> then later get the site to appear as if its hosted at the new domain
> name.
> 
> - Mike
> 
> ----- Original Message -----
> > From: "Tim Armstrong" <tim.g.armstrong at gmail.com>
> > To: "swift-devel" <swift-devel at ci.uchicago.edu>
> > Sent: Friday, May 24, 2013 5:01:33 PM
> > Subject: [Swift-devel] Domain name
> > 
> > 
> > 
> > 
> > Following up on the earlier discussion, swift-lang.org isn't taken.
> > 
> > 
> > - Tim
> > 
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> 


From skrieder at iit.edu  Wed Jun  5 13:48:45 2013
From: skrieder at iit.edu (Scott Krieder)
Date: Wed, 5 Jun 2013 13:48:45 -0500
Subject: [Swift-devel] Domain name
In-Reply-To: <1529864766.497163.1370457577538.JavaMail.root@mcs.anl.gov>
References: <etPan.51af80ba.4e6afb66.c4@Vytas-Surrogate.local>
	<1529864766.497163.1370457577538.JavaMail.root@mcs.anl.gov>
Message-ID: <CABp7gVzEjyHvbqpMx1yZwivKCyOfjCnsE5PnJPDPhCJup4xkaw@mail.gmail.com>

I think that either swiftlang.org or swift-lang.org sound the best. I have
setup domain forwarding for several of my GoDaddy domains using this guide:
http://support.godaddy.com/help/article/422/forwarding-or-masking-your-domain-name

If you'd like me to help with this at all just let me know.

-Scott


On Wed, Jun 5, 2013 at 1:39 PM, Michael Wilde <wilde at mcs.anl.gov> wrote:

> We're trying to deprecate the use of the name "SwiftScript". Its a bit
> hard to pronounce (rolls off the tongue poorly) and we want to be more
> selective in whether we pitch it as a scripting language or parallel
> programming language.
>
> We need to select one URL like http://swift-lang.org and all other URLs
> use should bring you to the site under this preferred URL.
>
> I agree with David on preferring swift-lang.org: it "looks" best, even if
> we occasionally have to point out the hyphen.
>
> Vytas: any advice on how best to do this and what to watch out for? Is it
> simply a matter of setting the site to redirect all entry points to one
> primary URL?
>
> - Mike
>
> ----- Original Message -----
> > From: "Vytas Cuplinskas" <vytas at ci.uchicago.edu>
> > To: "David Kelly" <davidk at ci.uchicago.edu>, "swift-devel" <
> swift-devel at ci.uchicago.edu>
> > Cc: "Michael Wilde" <wilde at mcs.anl.gov>
> > Sent: Wednesday, June 5, 2013 1:17:30 PM
> > Subject: Re: [Swift-devel] Domain name
> >
> >
> >
> > Just curious ? did you ever considered swiftscript.org?
> >
> >
> > Vytas
> >
> >
> >
> > P.S. if this is opening a Pandora's box please disregard
> >
> >
> >
> > On June 5, 2013 at 1:07:46 PM, David Kelly (davidk at ci.uchicago.edu)
> > wrote:
> >
> >
> >
> > Hello all,
> >
> >
> > Mike wanted me to email the list to see if there was a consensus on
> > what our primary domain name should be. I think the idea is to have
> > one domain name be the official site, then have the others redirect
> > there. The list we have:
> >
> >
> >
> > swiftlang.org
> > swiftlang.com
> > swiftlang.net
> > swift-lang.org
> > swift-lang.com
> > swift-lang.net
> >
> >
> > At the moment, swiftlang.org and swift-lang.org are both configured
> > to load the Swift page.
> >
> >
> > My preference is swift-lang.org.
> >
> >
> > David
> >
> >
> > ----- Original Message -----
> >
> >
> > From: "Michael Wilde" <wilde at mcs.anl.gov>
> > To: "swift-devel" <swift-devel at ci.uchicago.edu>
> > Sent: Friday, May 24, 2013 7:25:27 PM
> > Subject: Re: [Swift-devel] Domain name
> >
> > I got swiftlang.org and swift-lang.org
> >
> > We can initially forward them to the www.ci.uchicago.edu/swift site,
> > then later get the site to appear as if its hosted at the new domain
> > name.
> >
> > - Mike
> >
> > ----- Original Message -----
> > > From: "Tim Armstrong" <tim.g.armstrong at gmail.com>
> > > To: "swift-devel" <swift-devel at ci.uchicago.edu>
> > > Sent: Friday, May 24, 2013 5:01:33 PM
> > > Subject: [Swift-devel] Domain name
> > >
> > >
> > >
> > >
> > > Following up on the earlier discussion, swift-lang.org isn't taken.
> > >
> > >
> > > - Tim
> > >
> > > _______________________________________________
> > > Swift-devel mailing list
> > > Swift-devel at ci.uchicago.edu
> > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >
> >
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>



-- 
Scott J. Krieder
C: 419-685-0410
E: skrieder at iit.edu
http://datasys.cs.iit.edu/~skrieder/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130605/d005ed24/attachment.html>

From hategan at mcs.anl.gov  Wed Jun  5 13:51:28 2013
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 05 Jun 2013 11:51:28 -0700
Subject: [Swift-devel] Domain name
In-Reply-To: <1754623550.2636252.1370455666048.JavaMail.root@ci.uchicago.edu>
References: <1754623550.2636252.1370455666048.JavaMail.root@ci.uchicago.edu>
Message-ID: <1370458288.18532.0.camel@echo>

I like the .org ones.

Mihael

On Wed, 2013-06-05 at 13:07 -0500, David Kelly wrote:
> Hello all, 
> 
> 
> Mike wanted me to email the list to see if there was a consensus on what our primary domain name should be. I think the idea is to have one domain name be the official site, then have the others redirect there. The list we have: 
> 
> 
> 
> swiftlang.org 
> swiftlang.com 
> swiftlang.net 
> swift-lang.org 
> swift-lang.com 
> swift-lang.net 
> 
> 
> At the moment, swiftlang.org and swift-lang.org are both configured to load the Swift page. 
> 
> 
> My preference is swift-lang.org. 
> 
> 
> David 
> 
> 
> ----- Original Message -----
> 
> 
> From: "Michael Wilde" <wilde at mcs.anl.gov> 
> To: "swift-devel" <swift-devel at ci.uchicago.edu> 
> Sent: Friday, May 24, 2013 7:25:27 PM 
> Subject: Re: [Swift-devel] Domain name 
> 
> I got swiftlang.org and swift-lang.org 
> 
> We can initially forward them to the www.ci.uchicago.edu/swift site, then later get the site to appear as if its hosted at the new domain name. 
> 
> - Mike 
> 
> ----- Original Message ----- 
> > From: "Tim Armstrong" <tim.g.armstrong at gmail.com> 
> > To: "swift-devel" <swift-devel at ci.uchicago.edu> 
> > Sent: Friday, May 24, 2013 5:01:33 PM 
> > Subject: [Swift-devel] Domain name 
> > 
> > 
> > 
> > 
> > Following up on the earlier discussion, swift-lang.org isn't taken. 
> > 
> > 
> > - Tim 
> > 
> > _______________________________________________ 
> > Swift-devel mailing list 
> > Swift-devel at ci.uchicago.edu 
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel 
> > 
> _______________________________________________ 
> Swift-devel mailing list 
> Swift-devel at ci.uchicago.edu 
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel 
> 
> 
> 
> Hello all,
> 
> 
> Mike wanted me to email the list to see if there was a consensus on
> what our primary domain name should be. I think the idea is to have
> one domain name be the official site, then have the others redirect
> there. The list we have:
> 
> 
> swiftlang.org
> swiftlang.com
> swiftlang.net
> swift-lang.org
> swift-lang.com
> swift-lang.net
> 
> 
> At the moment, swiftlang.org and swift-lang.org are both configured to
> load the Swift page.
> 
> 
> My preference is swift-lang.org.
> 
> 
> David
> 
> 
> 
> ______________________________________________________________________
>         From: "Michael Wilde" <wilde at mcs.anl.gov>
>         To: "swift-devel" <swift-devel at ci.uchicago.edu>
>         Sent: Friday, May 24, 2013 7:25:27 PM
>         Subject: Re: [Swift-devel] Domain name
>         
>         I got swiftlang.org and swift-lang.org
>         
>         We can initially forward them to the www.ci.uchicago.edu/swift
>         site, then later get the site to appear as if its hosted at
>         the new domain name.
>         
>         - Mike
>         
>         ----- Original Message -----
>         > From: "Tim Armstrong" <tim.g.armstrong at gmail.com>
>         > To: "swift-devel" <swift-devel at ci.uchicago.edu>
>         > Sent: Friday, May 24, 2013 5:01:33 PM
>         > Subject: [Swift-devel] Domain name
>         > 
>         > 
>         > 
>         > 
>         > Following up on the earlier discussion, swift-lang.org isn't
>         taken.
>         > 
>         > 
>         > - Tim
>         > 
>         > _______________________________________________
>         > Swift-devel mailing list
>         > Swift-devel at ci.uchicago.edu
>         >
>         https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>         > 
>         _______________________________________________
>         Swift-devel mailing list
>         Swift-devel at ci.uchicago.edu
>         https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel




From ketancmaheshwari at gmail.com  Wed Jun  5 13:55:27 2013
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Wed, 5 Jun 2013 13:55:27 -0500
Subject: [Swift-devel] Domain name
In-Reply-To: <1754623550.2636252.1370455666048.JavaMail.root@ci.uchicago.edu>
References: <1192744849.2045978.1369441527231.JavaMail.root@mcs.anl.gov>
	<1754623550.2636252.1370455666048.JavaMail.root@ci.uchicago.edu>
Message-ID: <CAMUuvirc9b27_A+9fVRgZM86-rSTs3VCjxQwoMzcU2R5MRPjng@mail.gmail.com>

I like swift-lang.org:

Other similar ones on quick search yield:

Google's go:  http://golang.org
erlang:          http://www.erlang.org
scala:           http://www.scala-lang.org
haskell:        http://www.haskell.org

org seems to be the trend. with hyphen sounds clearer.



On Wed, Jun 5, 2013 at 1:07 PM, David Kelly <davidk at ci.uchicago.edu> wrote:

> Hello all,
>
> Mike wanted me to email the list to see if there was a consensus on what
> our primary domain name should be. I think the idea is to have one domain
> name be the official site, then have the others redirect there. The list we
> have:
>
> swiftlang.org
> swiftlang.com
> swiftlang.net
> swift-lang.org
> swift-lang.com
> swift-lang.net
>
> At the moment, swiftlang.org and swift-lang.org are both configured to
> load the Swift page.
>
> My preference is swift-lang.org.
>
> David
>
> ------------------------------
>
> *From: *"Michael Wilde" <wilde at mcs.anl.gov>
> *To: *"swift-devel" <swift-devel at ci.uchicago.edu>
> *Sent: *Friday, May 24, 2013 7:25:27 PM
> *Subject: *Re: [Swift-devel] Domain name
>
>
> I got swiftlang.org and swift-lang.org
>
> We can initially forward them to the www.ci.uchicago.edu/swift site, then
> later get the site to appear as if its hosted at the new domain name.
>
> - Mike
>
> ----- Original Message -----
> > From: "Tim Armstrong" <tim.g.armstrong at gmail.com>
> > To: "swift-devel" <swift-devel at ci.uchicago.edu>
> > Sent: Friday, May 24, 2013 5:01:33 PM
> > Subject: [Swift-devel] Domain name
> >
> >
> >
> >
> > Following up on the earlier discussion, swift-lang.org isn't taken.
> >
> >
> > - Tim
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>


-- 
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130605/9099a427/attachment.html>

From wilde at mcs.anl.gov  Wed Jun  5 14:08:19 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 5 Jun 2013 14:08:19 -0500 (CDT)
Subject: [Swift-devel] Domain name
In-Reply-To: <CAMUuvirc9b27_A+9fVRgZM86-rSTs3VCjxQwoMzcU2R5MRPjng@mail.gmail.com>
Message-ID: <455972880.499924.1370459299579.JavaMail.root@mcs.anl.gov>

I apologize for the confusion: .org was the only root domain being considered.

The choice to be made is swift-lang.org vs. swiftlang.org for what we publicize.

So far there are 3 votes for swift-lang and 1 for either. Any more votes?

The other question was about forwarding all alias names to the single publicized URL. That seems to be answered now: Google provides tools to consolidate the chosen name, and user bookmarks and old references will all get forwarded to that name by our server.

Once this is all set up we can start using and publicizing the new name.

- Mike

----- Original Message -----
> From: "Ketan Maheshwari" <ketancmaheshwari at gmail.com>
> To: "David Kelly" <davidk at ci.uchicago.edu>
> Cc: "swift-devel" <swift-devel at ci.uchicago.edu>, "Vytas Cuplinskas" <vytas at ci.uchicago.edu>
> Sent: Wednesday, June 5, 2013 1:55:27 PM
> Subject: Re: [Swift-devel] Domain name
> 
> 
> 
> I like swift-lang.org :
> 
> 
> Other similar ones on quick search yield:
> 
> 
> Google's go: http://golang.org
> erlang: http://www.erlang.org
> scala: http://www.scala-lang.org
> haskell: http://www.haskell.org
> 
> 
> org seems to be the trend. with hyphen sounds clearer.
> 
> 
> 
> 
> 
> On Wed, Jun 5, 2013 at 1:07 PM, David Kelly < davidk at ci.uchicago.edu
> > wrote:
> 
> 
> 
> 
> Hello all,
> 
> 
> Mike wanted me to email the list to see if there was a consensus on
> what our primary domain name should be. I think the idea is to have
> one domain name be the official site, then have the others redirect
> there. The list we have:
> 
> 
> 
> swiftlang.org
> swiftlang.com
> swiftlang.net
> swift-lang.org
> swift-lang.com
> swift-lang.net
> 
> 
> At the moment, swiftlang.org and swift-lang.org are both configured
> to load the Swift page.
> 
> 
> My preference is swift-lang.org .
> 
> 
> David
> 
> 
> 
> 
> 
> From: "Michael Wilde" < wilde at mcs.anl.gov >
> To: "swift-devel" < swift-devel at ci.uchicago.edu >
> Sent: Friday, May 24, 2013 7:25:27 PM
> Subject: Re: [Swift-devel] Domain name
> 
> 
> 
> I got swiftlang.org and swift-lang.org
> 
> We can initially forward them to the www.ci.uchicago.edu/swift site,
> then later get the site to appear as if its hosted at the new domain
> name.
> 
> - Mike
> 
> ----- Original Message -----
> > From: "Tim Armstrong" < tim.g.armstrong at gmail.com >
> > To: "swift-devel" < swift-devel at ci.uchicago.edu >
> > Sent: Friday, May 24, 2013 5:01:33 PM
> > Subject: [Swift-devel] Domain name
> > 
> > 
> > 
> > 
> > Following up on the earlier discussion, swift-lang.org isn't taken.
> > 
> > 
> > - Tim
> > 
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> 
> 
> 
> 
> --
> Ketan
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 


From dsk at ci.uchicago.edu  Wed Jun  5 14:10:09 2013
From: dsk at ci.uchicago.edu (Daniel S. Katz)
Date: Wed, 5 Jun 2013 15:10:09 -0400
Subject: [Swift-devel] Domain name
In-Reply-To: <455972880.499924.1370459299579.JavaMail.root@mcs.anl.gov>
References: <455972880.499924.1370459299579.JavaMail.root@mcs.anl.gov>
Message-ID: <0F03DACF-8E6B-4413-A6CB-1AC1B72A8D35@ci.uchicago.edu>

with -


On Jun 5, 2013, at 3:08 PM, Michael Wilde <wilde at mcs.anl.gov> wrote:

> I apologize for the confusion: .org was the only root domain being considered.
> 
> The choice to be made is swift-lang.org vs. swiftlang.org for what we publicize.
> 
> So far there are 3 votes for swift-lang and 1 for either. Any more votes?
> 
> The other question was about forwarding all alias names to the single publicized URL. That seems to be answered now: Google provides tools to consolidate the chosen name, and user bookmarks and old references will all get forwarded to that name by our server.
> 
> Once this is all set up we can start using and publicizing the new name.
> 
> - Mike
> 
> ----- Original Message -----
>> From: "Ketan Maheshwari" <ketancmaheshwari at gmail.com>
>> To: "David Kelly" <davidk at ci.uchicago.edu>
>> Cc: "swift-devel" <swift-devel at ci.uchicago.edu>, "Vytas Cuplinskas" <vytas at ci.uchicago.edu>
>> Sent: Wednesday, June 5, 2013 1:55:27 PM
>> Subject: Re: [Swift-devel] Domain name
>> 
>> 
>> 
>> I like swift-lang.org :
>> 
>> 
>> Other similar ones on quick search yield:
>> 
>> 
>> Google's go: http://golang.org
>> erlang: http://www.erlang.org
>> scala: http://www.scala-lang.org
>> haskell: http://www.haskell.org
>> 
>> 
>> org seems to be the trend. with hyphen sounds clearer.
>> 
>> 
>> 
>> 
>> 
>> On Wed, Jun 5, 2013 at 1:07 PM, David Kelly < davidk at ci.uchicago.edu
>>> wrote:
>> 
>> 
>> 
>> 
>> Hello all,
>> 
>> 
>> Mike wanted me to email the list to see if there was a consensus on
>> what our primary domain name should be. I think the idea is to have
>> one domain name be the official site, then have the others redirect
>> there. The list we have:
>> 
>> 
>> 
>> swiftlang.org
>> swiftlang.com
>> swiftlang.net
>> swift-lang.org
>> swift-lang.com
>> swift-lang.net
>> 
>> 
>> At the moment, swiftlang.org and swift-lang.org are both configured
>> to load the Swift page.
>> 
>> 
>> My preference is swift-lang.org .
>> 
>> 
>> David
>> 
>> 
>> 
>> 
>> 
>> From: "Michael Wilde" < wilde at mcs.anl.gov >
>> To: "swift-devel" < swift-devel at ci.uchicago.edu >
>> Sent: Friday, May 24, 2013 7:25:27 PM
>> Subject: Re: [Swift-devel] Domain name
>> 
>> 
>> 
>> I got swiftlang.org and swift-lang.org
>> 
>> We can initially forward them to the www.ci.uchicago.edu/swift site,
>> then later get the site to appear as if its hosted at the new domain
>> name.
>> 
>> - Mike
>> 
>> ----- Original Message -----
>>> From: "Tim Armstrong" < tim.g.armstrong at gmail.com >
>>> To: "swift-devel" < swift-devel at ci.uchicago.edu >
>>> Sent: Friday, May 24, 2013 5:01:33 PM
>>> Subject: [Swift-devel] Domain name
>>> 
>>> 
>>> 
>>> 
>>> Following up on the earlier discussion, swift-lang.org isn't taken.
>>> 
>>> 
>>> - Tim
>>> 
>>> _______________________________________________
>>> Swift-devel mailing list
>>> Swift-devel at ci.uchicago.edu
>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>> 
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>> 
>> 
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>> 
>> 
>> 
>> 
>> 
>> --
>> Ketan
>> 
>> 
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-- 
Daniel S. Katz
University of Chicago
(773) 834-7186 (voice)
(773) 834-6818 (fax)
d.katz at ieee.org or dsk at ci.uchicago.edu
http://www.ci.uchicago.edu/~dsk/




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130605/a1431678/attachment.html>

From wozniak at mcs.anl.gov  Wed Jun  5 14:27:49 2013
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Wed, 05 Jun 2013 14:27:49 -0500
Subject: [Swift-devel] Domain name
In-Reply-To: <CABp7gVzEjyHvbqpMx1yZwivKCyOfjCnsE5PnJPDPhCJup4xkaw@mail.gmail.com>
References: <etPan.51af80ba.4e6afb66.c4@Vytas-Surrogate.local>
	<1529864766.497163.1370457577538.JavaMail.root@mcs.anl.gov>
	<CABp7gVzEjyHvbqpMx1yZwivKCyOfjCnsE5PnJPDPhCJup4xkaw@mail.gmail.com>
Message-ID: <51AF9135.3090009@mcs.anl.gov>


I also like http://swift-lang.org

On 06/05/2013 01:48 PM, Scott Krieder wrote:
> I think that either swiftlang.org <http://swiftlang.org> or 
> swift-lang.org <http://swift-lang.org> sound the best. I have setup 
> domain forwarding for several of my GoDaddy domains using this guide:
> http://support.godaddy.com/help/article/422/forwarding-or-masking-your-domain-name
>
> If you'd like me to help with this at all just let me know.
>
> -Scott
>
>
> On Wed, Jun 5, 2013 at 1:39 PM, Michael Wilde <wilde at mcs.anl.gov 
> <mailto:wilde at mcs.anl.gov>> wrote:
>
>     We're trying to deprecate the use of the name "SwiftScript". Its a
>     bit hard to pronounce (rolls off the tongue poorly) and we want to
>     be more selective in whether we pitch it as a scripting language
>     or parallel programming language.
>
>     We need to select one URL like http://swift-lang.org and all other
>     URLs use should bring you to the site under this preferred URL.
>
>     I agree with David on preferring swift-lang.org
>     <http://swift-lang.org>: it "looks" best, even if we occasionally
>     have to point out the hyphen.
>
>     Vytas: any advice on how best to do this and what to watch out
>     for? Is it simply a matter of setting the site to redirect all
>     entry points to one primary URL?
>
>     - Mike
>
>     ----- Original Message -----
>     > From: "Vytas Cuplinskas" <vytas at ci.uchicago.edu <mailto:vytas at ci.uchicago.edu>>
>     > To: "David Kelly" <davidk at ci.uchicago.edu
>     <mailto:davidk at ci.uchicago.edu>>, "swift-devel"
>     <swift-devel at ci.uchicago.edu <mailto:swift-devel at ci.uchicago.edu>>
>     > Cc: "Michael Wilde" <wilde at mcs.anl.gov <mailto:wilde at mcs.anl.gov>>
>     > Sent: Wednesday, June 5, 2013 1:17:30 PM
>     > Subject: Re: [Swift-devel] Domain name
>     >
>     >
>     >
>     > Just curious -- did you ever considered swiftscript.org
>     <http://swiftscript.org>?
>     >
>     >
>     > Vytas
>     >
>     >
>     >
>     > P.S. if this is opening a Pandora's box please disregard
>     >
>     >
>     >
>     > On June 5, 2013 at 1:07:46 PM, David Kelly
>     (davidk at ci.uchicago.edu <mailto:davidk at ci.uchicago.edu>)
>     > wrote:
>     >
>     >
>     >
>     > Hello all,
>     >
>     >
>     > Mike wanted me to email the list to see if there was a consensus on
>     > what our primary domain name should be. I think the idea is to have
>     > one domain name be the official site, then have the others redirect
>     > there. The list we have:
>     >
>     >
>     >
>     > swiftlang.org <http://swiftlang.org>
>     > swiftlang.com <http://swiftlang.com>
>     > swiftlang.net <http://swiftlang.net>
>     > swift-lang.org <http://swift-lang.org>
>     > swift-lang.com <http://swift-lang.com>
>     > swift-lang.net <http://swift-lang.net>
>     >
>     >
>     > At the moment, swiftlang.org <http://swiftlang.org> and
>     swift-lang.org <http://swift-lang.org> are both configured
>     > to load the Swift page.
>     >
>     >
>     > My preference is swift-lang.org <http://swift-lang.org>.
>     >
>     >
>     > David
>     >
>     >
>     > ----- Original Message -----
>     >
>     >
>     > From: "Michael Wilde" <wilde at mcs.anl.gov <mailto:wilde at mcs.anl.gov>>
>     > To: "swift-devel" <swift-devel at ci.uchicago.edu
>     <mailto:swift-devel at ci.uchicago.edu>>
>     > Sent: Friday, May 24, 2013 7:25:27 PM
>     > Subject: Re: [Swift-devel] Domain name
>     >
>     > I got swiftlang.org <http://swiftlang.org> and swift-lang.org
>     <http://swift-lang.org>
>     >
>     > We can initially forward them to the www.ci.uchicago.edu/swift
>     <http://www.ci.uchicago.edu/swift> site,
>     > then later get the site to appear as if its hosted at the new domain
>     > name.
>     >
>     > - Mike
>     >
>     > ----- Original Message -----
>     > > From: "Tim Armstrong" <tim.g.armstrong at gmail.com
>     <mailto:tim.g.armstrong at gmail.com>>
>     > > To: "swift-devel" <swift-devel at ci.uchicago.edu
>     <mailto:swift-devel at ci.uchicago.edu>>
>     > > Sent: Friday, May 24, 2013 5:01:33 PM
>     > > Subject: [Swift-devel] Domain name
>     > >
>     > >
>     > >
>     > >
>     > > Following up on the earlier discussion, swift-lang.org
>     <http://swift-lang.org> isn't taken.
>     > >
>     > >
>     > > - Tim
>     > >
>     > > _______________________________________________
>     > > Swift-devel mailing list
>     > > Swift-devel at ci.uchicago.edu <mailto:Swift-devel at ci.uchicago.edu>
>     > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>     > >
>     > _______________________________________________
>     > Swift-devel mailing list
>     > Swift-devel at ci.uchicago.edu <mailto:Swift-devel at ci.uchicago.edu>
>     > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>     >
>     >
>     _______________________________________________
>     Swift-devel mailing list
>     Swift-devel at ci.uchicago.edu <mailto:Swift-devel at ci.uchicago.edu>
>     https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>
>
>
> -- 
> Scott J. Krieder
> C: 419-685-0410
> E: skrieder at iit.edu <mailto:skrieder at iit.edu>
> http://datasys.cs.iit.edu/~skrieder/ 
> <http://datasys.cs.iit.edu/%7Eskrieder/>
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel


-- 
Justin M Wozniak

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130605/0ff01b4e/attachment.html>

From yadudoc1729 at gmail.com  Wed Jun  5 14:36:14 2013
From: yadudoc1729 at gmail.com (Yadu Nand)
Date: Wed, 5 Jun 2013 14:36:14 -0500
Subject: [Swift-devel] Domain name
In-Reply-To: <51AF9135.3090009@mcs.anl.gov>
References: <etPan.51af80ba.4e6afb66.c4@Vytas-Surrogate.local>
	<1529864766.497163.1370457577538.JavaMail.root@mcs.anl.gov>
	<CABp7gVzEjyHvbqpMx1yZwivKCyOfjCnsE5PnJPDPhCJup4xkaw@mail.gmail.com>
	<51AF9135.3090009@mcs.anl.gov>
Message-ID: <CANa904=+Oe0D0p1WU9YN9jhxjfPj0oWri-b2PONSJyLXAio61g@mail.gmail.com>

+1 for swift-lang.org

-Yadu

On Wed, Jun 5, 2013 at 2:27 PM, Justin M Wozniak <wozniak at mcs.anl.gov> wrote:
>
> I also like http://swift-lang.org
>
> On 06/05/2013 01:48 PM, Scott Krieder wrote:
>
> I think that either swiftlang.org or swift-lang.org sound the best. I have
> setup domain forwarding for several of my GoDaddy domains using this guide:
> http://support.godaddy.com/help/article/422/forwarding-or-masking-your-domain-name
>
> If you'd like me to help with this at all just let me know.
>
> -Scott
>
>
> On Wed, Jun 5, 2013 at 1:39 PM, Michael Wilde <wilde at mcs.anl.gov> wrote:
>>
>> We're trying to deprecate the use of the name "SwiftScript". Its a bit
>> hard to pronounce (rolls off the tongue poorly) and we want to be more
>> selective in whether we pitch it as a scripting language or parallel
>> programming language.
>>
>> We need to select one URL like http://swift-lang.org and all other URLs
>> use should bring you to the site under this preferred URL.
>>
>> I agree with David on preferring swift-lang.org: it "looks" best, even if
>> we occasionally have to point out the hyphen.
>>
>> Vytas: any advice on how best to do this and what to watch out for? Is it
>> simply a matter of setting the site to redirect all entry points to one
>> primary URL?
>>
>> - Mike
>>
>> ----- Original Message -----
>> > From: "Vytas Cuplinskas" <vytas at ci.uchicago.edu>
>> > To: "David Kelly" <davidk at ci.uchicago.edu>, "swift-devel"
>> > <swift-devel at ci.uchicago.edu>
>> > Cc: "Michael Wilde" <wilde at mcs.anl.gov>
>> > Sent: Wednesday, June 5, 2013 1:17:30 PM
>> > Subject: Re: [Swift-devel] Domain name
>> >
>> >
>> >
>> > Just curious ? did you ever considered swiftscript.org?
>> >
>> >
>> > Vytas
>> >
>> >
>> >
>> > P.S. if this is opening a Pandora's box please disregard
>> >
>> >
>> >
>> > On June 5, 2013 at 1:07:46 PM, David Kelly (davidk at ci.uchicago.edu)
>> > wrote:
>> >
>> >
>> >
>> > Hello all,
>> >
>> >
>> > Mike wanted me to email the list to see if there was a consensus on
>> > what our primary domain name should be. I think the idea is to have
>> > one domain name be the official site, then have the others redirect
>> > there. The list we have:
>> >
>> >
>> >
>> > swiftlang.org
>> > swiftlang.com
>> > swiftlang.net
>> > swift-lang.org
>> > swift-lang.com
>> > swift-lang.net
>> >
>> >
>> > At the moment, swiftlang.org and swift-lang.org are both configured
>> > to load the Swift page.
>> >
>> >
>> > My preference is swift-lang.org.
>> >
>> >
>> > David
>> >
>> >
>> > ----- Original Message -----
>> >
>> >
>> > From: "Michael Wilde" <wilde at mcs.anl.gov>
>> > To: "swift-devel" <swift-devel at ci.uchicago.edu>
>> > Sent: Friday, May 24, 2013 7:25:27 PM
>> > Subject: Re: [Swift-devel] Domain name
>> >
>> > I got swiftlang.org and swift-lang.org
>> >
>> > We can initially forward them to the www.ci.uchicago.edu/swift site,
>> > then later get the site to appear as if its hosted at the new domain
>> > name.
>> >
>> > - Mike
>> >
>> > ----- Original Message -----
>> > > From: "Tim Armstrong" <tim.g.armstrong at gmail.com>
>> > > To: "swift-devel" <swift-devel at ci.uchicago.edu>
>> > > Sent: Friday, May 24, 2013 5:01:33 PM
>> > > Subject: [Swift-devel] Domain name
>> > >
>> > >
>> > >
>> > >
>> > > Following up on the earlier discussion, swift-lang.org isn't taken.
>> > >
>> > >
>> > > - Tim
>> > >
>> > > _______________________________________________
>> > > Swift-devel mailing list
>> > > Swift-devel at ci.uchicago.edu
>> > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>> > >
>> > _______________________________________________
>> > Swift-devel mailing list
>> > Swift-devel at ci.uchicago.edu
>> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>> >
>> >
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>
>
>
> --
> Scott J. Krieder
> C: 419-685-0410
> E: skrieder at iit.edu
> http://datasys.cs.iit.edu/~skrieder/
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>
>
> --
> Justin M Wozniak
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>



-- 
Yadu Nand B


From wilde at mcs.anl.gov  Wed Jun  5 14:46:47 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 5 Jun 2013 14:46:47 -0500 (CDT)
Subject: [Swift-devel] Domain name
In-Reply-To: <CANa904=+Oe0D0p1WU9YN9jhxjfPj0oWri-b2PONSJyLXAio61g@mail.gmail.com>
Message-ID: <1772215684.504763.1370461607664.JavaMail.root@mcs.anl.gov>

David, given what looks like a majority in favor of swift-lang.org, can you make the site redirect all 16 new combinations to this? Ideally including if someone tacks on "/swift".

Then when the Google settings are ready we can also redirect www.ci.uchicago.edu/swift to swift-lang.org. To get Google to consolidate the current URL to the new one might require action by the owner of ci.uchicago.edu domain.

Lastly, there are some URLs (I think from Google searches) that still bring you to the old ("maroon") site somehow. We should see if we can find those and redirect them to the right place in the new site or at least to the new primary URL.

- Mike


----- Original Message -----
> From: "Yadu Nand" <yadudoc1729 at gmail.com>
> To: "Justin M Wozniak" <wozniak at mcs.anl.gov>
> Cc: swift-devel at ci.uchicago.edu
> Sent: Wednesday, June 5, 2013 2:36:14 PM
> Subject: Re: [Swift-devel] Domain name
> 
> +1 for swift-lang.org
> 
> -Yadu
> 
> On Wed, Jun 5, 2013 at 2:27 PM, Justin M Wozniak
> <wozniak at mcs.anl.gov> wrote:
> >
> > I also like http://swift-lang.org
> >
> > On 06/05/2013 01:48 PM, Scott Krieder wrote:
> >
> > I think that either swiftlang.org or swift-lang.org sound the best.
> > I have
> > setup domain forwarding for several of my GoDaddy domains using
> > this guide:
> > http://support.godaddy.com/help/article/422/forwarding-or-masking-your-domain-name
> >
> > If you'd like me to help with this at all just let me know.
> >
> > -Scott
> >
> >
> > On Wed, Jun 5, 2013 at 1:39 PM, Michael Wilde <wilde at mcs.anl.gov>
> > wrote:
> >>
> >> We're trying to deprecate the use of the name "SwiftScript". Its a
> >> bit
> >> hard to pronounce (rolls off the tongue poorly) and we want to be
> >> more
> >> selective in whether we pitch it as a scripting language or
> >> parallel
> >> programming language.
> >>
> >> We need to select one URL like http://swift-lang.org and all other
> >> URLs
> >> use should bring you to the site under this preferred URL.
> >>
> >> I agree with David on preferring swift-lang.org: it "looks" best,
> >> even if
> >> we occasionally have to point out the hyphen.
> >>
> >> Vytas: any advice on how best to do this and what to watch out
> >> for? Is it
> >> simply a matter of setting the site to redirect all entry points
> >> to one
> >> primary URL?
> >>
> >> - Mike
> >>
> >> ----- Original Message -----
> >> > From: "Vytas Cuplinskas" <vytas at ci.uchicago.edu>
> >> > To: "David Kelly" <davidk at ci.uchicago.edu>, "swift-devel"
> >> > <swift-devel at ci.uchicago.edu>
> >> > Cc: "Michael Wilde" <wilde at mcs.anl.gov>
> >> > Sent: Wednesday, June 5, 2013 1:17:30 PM
> >> > Subject: Re: [Swift-devel] Domain name
> >> >
> >> >
> >> >
> >> > Just curious ? did you ever considered swiftscript.org?
> >> >
> >> >
> >> > Vytas
> >> >
> >> >
> >> >
> >> > P.S. if this is opening a Pandora's box please disregard
> >> >
> >> >
> >> >
> >> > On June 5, 2013 at 1:07:46 PM, David Kelly
> >> > (davidk at ci.uchicago.edu)
> >> > wrote:
> >> >
> >> >
> >> >
> >> > Hello all,
> >> >
> >> >
> >> > Mike wanted me to email the list to see if there was a consensus
> >> > on
> >> > what our primary domain name should be. I think the idea is to
> >> > have
> >> > one domain name be the official site, then have the others
> >> > redirect
> >> > there. The list we have:
> >> >
> >> >
> >> >
> >> > swiftlang.org
> >> > swiftlang.com
> >> > swiftlang.net
> >> > swift-lang.org
> >> > swift-lang.com
> >> > swift-lang.net
> >> >
> >> >
> >> > At the moment, swiftlang.org and swift-lang.org are both
> >> > configured
> >> > to load the Swift page.
> >> >
> >> >
> >> > My preference is swift-lang.org.
> >> >
> >> >
> >> > David
> >> >
> >> >
> >> > ----- Original Message -----
> >> >
> >> >
> >> > From: "Michael Wilde" <wilde at mcs.anl.gov>
> >> > To: "swift-devel" <swift-devel at ci.uchicago.edu>
> >> > Sent: Friday, May 24, 2013 7:25:27 PM
> >> > Subject: Re: [Swift-devel] Domain name
> >> >
> >> > I got swiftlang.org and swift-lang.org
> >> >
> >> > We can initially forward them to the www.ci.uchicago.edu/swift
> >> > site,
> >> > then later get the site to appear as if its hosted at the new
> >> > domain
> >> > name.
> >> >
> >> > - Mike
> >> >
> >> > ----- Original Message -----
> >> > > From: "Tim Armstrong" <tim.g.armstrong at gmail.com>
> >> > > To: "swift-devel" <swift-devel at ci.uchicago.edu>
> >> > > Sent: Friday, May 24, 2013 5:01:33 PM
> >> > > Subject: [Swift-devel] Domain name
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > Following up on the earlier discussion, swift-lang.org isn't
> >> > > taken.
> >> > >
> >> > >
> >> > > - Tim
> >> > >
> >> > > _______________________________________________
> >> > > Swift-devel mailing list
> >> > > Swift-devel at ci.uchicago.edu
> >> > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >> > >
> >> > _______________________________________________
> >> > Swift-devel mailing list
> >> > Swift-devel at ci.uchicago.edu
> >> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >> >
> >> >
> >> _______________________________________________
> >> Swift-devel mailing list
> >> Swift-devel at ci.uchicago.edu
> >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >
> >
> >
> >
> > --
> > Scott J. Krieder
> > C: 419-685-0410
> > E: skrieder at iit.edu
> > http://datasys.cs.iit.edu/~skrieder/
> >
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >
> >
> >
> > --
> > Justin M Wozniak
> >
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >
> 
> 
> 
> --
> Yadu Nand B
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 


From wozniak at mcs.anl.gov  Fri Jun  7 14:39:56 2013
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Fri, 07 Jun 2013 14:39:56 -0500
Subject: [Swift-devel] Swift team meeting today - 2PM central - Agenda
In-Reply-To: <324747402.1303584.1370627520939.JavaMail.root@mcs.anl.gov>
References: <324747402.1303584.1370627520939.JavaMail.root@mcs.anl.gov>
Message-ID: <51B2370C.2010804@mcs.anl.gov>

On 06/07/2013 12:52 PM, Michael Wilde wrote:

Ketan is working on two power grid apps and had some minor questions 
about Coasters for Mihael.

Yadu is working on the multi-site test suite which is itself implemented 
as a Swift script.

Scott is looking at CUDA BLAS, OOPS, and mdproxy apps for GPU.

Justin added some minor new syntax in Swift/T. Made significant 
additions to the Swift/T web materials.  Provided Scott with an 
eigenvalue code based on BLAS compatible with our model for Swift/T and 
the GPU.  Lots of conversation with the RDCEP and APS app cases.  For 
RDCEP, now have a better sense of how to validate and compare the 
Swift/T implementation to the previous pure Fortran/MPI app.  For APS, 
have a better sense of overall workflow and some ideas on where to find 
concurrency to target with Swift/T.  Also working with Ketan on Swift 
storage access on the cloud, focusing on the Chirp experiments.

Tim is working on fixing up some remaining broken Swift/T tests.

Mihael is looking at some coasters data movement timeout errors reported 
by Yadu.

> o David outline strategy for swift command rework (esp. tc wildcards)
David is investigating pre-processing the Swift/K command line with 
Perl.  Mihael suggested some improvements to automatic app configuration 
without tc.data .
> o David: outline priority for corresponding User Guide revamp
David will have more time shortly to work on this.
> o David: plans to phase in swift-lang.org and SWFT email list
David is in touch with David Forero about this.
> o Mihael: help Yadu and David start documenting the protocol for coasters in prep for coaster provider staging file caching feature work
Done somewhere- Mihael will send Yadu and David the link.
> o Yadu: review your observed Swift weaknesses and discuss & file (in bugz) suggestions to address each
> o Justin, Tim: next steps in Coaster integration
Yadu has the coasters C client running.  Discussed a list of next steps 
with Justin, including wrapping the client in SWIG and running it from 
Swift/T.  Justin provided code samples for reference.
> o Ketan: perhaps discuss next steps for making AWS resources easy to use by Swift end users
Ketan is doing this for the power grid app people.
> o All: any other topics
>
> Please report anything notable to swift-devel
>
> Thanks, All, and have a great weekend.
>
> - Mike
>
>


-- 
Justin M Wozniak



From wilde at mcs.anl.gov  Fri Jun  7 21:48:52 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Fri, 7 Jun 2013 21:48:52 -0500
Subject: [Swift-devel] [swift-bugs] [Bug 1014] mpi jobs fail on stampede
In-Reply-To: <20130608020408.C8745563CE@wind.mcs.anl.gov>
References: <bug-1014-54@http.bugzilla.mcs.anl.gov/swift/>
	<20130608020408.C8745563CE@wind.mcs.anl.gov>
Message-ID: <CAF1H9xqT0bzJ9Ccot+HE-+vMFz18=aH0EHEdg3k6BdwAkJgGGw@mail.gmail.com>

This really looks like the mpich2 fd0 problem.
Please try the " mpiexec < /dev/null fix first. This is well tested.

On 6/7/13, bugzilla-daemon at mcs.anl.gov <bugzilla-daemon at mcs.anl.gov> wrote:
> https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1014
>
>
>
>
>
> --- Comment #7 from Mihael Hategan <hategan at mcs.anl.gov>  2013-06-07
> 21:04:08 ---
> A google search suggests that the HYD messages are cleanup messages and
> happen
> after the user app exits.
>
> So I'd check if this is not a path issue (put
> /home1/01739/ketan/bin/mpitest.sh
> in the slurm script).
>
> I would also put some debugging lines in mpitest.sh (e.g. touch
> /shared/fs/path/mpitest_was_here) to see if it's started by swift.
>
> --
> Configure bugmail:
> https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email
> ------- You are receiving this mail because: -------
> You are watching all bug changes.
> _______________________________________________
> swift-bugs mailing list
> swift-bugs at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-bugs
>

-- 
Sent from my mobile device


From iraicu at cs.iit.edu  Sat Jun  8 08:30:40 2013
From: iraicu at cs.iit.edu (Ioan Raicu)
Date: Sat, 08 Jun 2013 08:30:40 -0500
Subject: [Swift-devel] CFP: ACM Many-Task Computing on Clouds, Grids,
 and Supercomputers (MTAGS) 2013 @ IEEE/ACM Supercomputing/SC 2013
Message-ID: <51B33200.6030000@cs.iit.edu>

CALL FOR PAPERS

6th Workshop on Many-Task Computing on Clouds, Grids, and Supercomputers (MTAGS) 2013
http://datasys.cs.iit.edu/events/MTAGS13/

Co-located with IEEE/ACM Supercomputing/SC 2013
Denver Colorado -- November 17th, 2013

Overview
------------------------------------------------------------------------------
The 6th workshop on Many-Task Computing on Grids and Supercomputers
(MTAGS) will provide the scientific community a dedicated forum for
presenting new research, development, and deployment efforts of
large-scale many-task computing (MTC) applications on large scale
clusters, Grids, Supercomputers, and Cloud Computing
infrastructure. MTC, the theme of the workshop encompasses loosely
coupled applications, which are generally composed of many tasks (both
independent and dependent tasks) to achieve some larger application
goal.  This workshop will cover challenges that can hamper efficiency
and utilization in running applications on large-scale systems, such
as local resource manager scalability and granularity, efficient
utilization of raw hardware, parallel file system contention and
scalability, data management, I/O management, reliability at scale,
and application scalability. We welcome paper submissions on all
theoretical, simulations, and systems topics related to MTC, but we
give special consideration to papers addressing petascale to exascale
challenges. Papers will be peer-reviewed, and accepted papers will be
published in the workshop proceedings as part of the ACM digital
library (pending approval). The workshop will be co-located with the
IEEE/ACM Supercomputing 2013 Conference in Denver Colorado on November
18th, 2013.

For more information, please see
http://datasys.cs.iit.edu/events/MTAGS13/. For more information on
past workshops, please see MTAGS12, MTAGS11, MTAGS10, MTAGS09, and
MTAGS08. We also ran a Special Issue on Many-Task Computing in the
IEEE Transactions on Parallel and Distributed Systems (TPDS) which has
appeared in June 2011; the proceedings can be found online at
http://www.computer.org/portal/web/csdl/abs/trans/td/2011/06/ttd201106toc.htm. We,
the workshop organizers, also published a highly relevant paper that
defines Many-Task Computing which was published in MTAGS08, titled
Many-Task Computing for Grids and Supercomputers; we encourage
potential authors to read this paper, and to clearly articulate in
your paper submissions how your papers are related to Many-Task
Computing.

Topics
------------------------------------------------------------------------------
We invite the submission of original work that is related to the
topics below. The papers should be 6 pages, including all figures and
references. We aim to cover topics related to Many-Task Computing on
each of the three major distributed systems paradigms, Cloud
Computing, Grid Computing and Supercomputing. Topics of interest
include:
      Compute Resource Management
         Scheduling
         Job execution frameworks
         Local resource manager extensions
         Performance evaluation of resource managers in use on large scale systems
         Dynamic resource provisioning
         Techniques to manage many-core resources and/or GPUs
         Challenges and opportunities in running many-task workloads on HPC systems
         Challenges and opportunities in running many-task workloads on Cloud infrastructure
      Storage architectures and implementations
         Distributed file systems
         Parallel file systems
         Distributed metadata management
         Content distribution systems for large data
         Data caching frameworks and techniques
         Data management within and across data centers
         Data-aware scheduling
         Data-intensive computing applications
         Eventual-consistency storage usage and management
      Programming models and tools
         MapReduce and its generalizations
         Many-task computing middleware and applications
         Parallel programming frameworks
         Ensemble MPI techniques and frameworks
         Service-oriented science applications
      Large-Scale Workflow Systems
         Workflow system performance and scalability analysis
         Scalability of workflow systems
         Workflow infrastructure and e-Science middleware
         Programming paradigms and models
      Large-Scale Many-Task Applications
         High-throughput computing (HTC) applications
         Data-intensive applications
         Quasi-supercomputing applications, deployments, and experiences
         Performance Evaluation
      Performance evaluation
         Real systems
         Simulations
         Reliability of large systems
     How MTC Addresses Challenges of Petascale and Exascale Computing
         Concurrency & Programmability
         I/O & Memory
         Energy
         Resilience
         Heterogeneity

Important Dates
------------------------------------------------------------------------------
     Paper submission: September 1, 2013
     Acceptance notification: October 13, 2013
     Final papers due: November 10th, 2013

Paper Submission
------------------------------------------------------------------------------
Authors are invited to submit papers with unpublished, original work
of not more than 6 pages of double column text using single spaced 10
point size on 8.5 x 11 inch pages, as per ACM 8.5 x 11 manuscript
guidelines; document templates can be found at
http://www.acm.org/sigs/publications/proceedings-templates. The final
6 page papers in PDF format must be submitted online at
https://cmt.research.microsoft.com/MTAGS2013/ before the deadline of
September 1st, 2013 at 11:59PM PST. Papers will be peer-reviewed, and
accepted papers will be published in the workshop proceedings as part
of the ACM digital library (pending approval). Notifications of the
paper decisions will be sent out by October 13th, 2011. Selected
excellent work may be eligible for additional post-conference
publication as journal articles. Submission implies the willingness of
at least one of the authors to register and present the paper. For
more information, please see http://datasys.cs.iit.edu/events/MTAGS13

Organization
------------------------------------------------------------------------------
General Chairs
     Ioan Raicu, Illinois Institute of Technology & Argonne National Laboratory, USA
     Ian Foster, University of Chicago & Argonne National Laboratory, USA
     Yong Zhao, University of Electronic Science and Technology of China, China
     Justin Wozniak, Argonne National Laboratory, USA

Steering Committee
     David Abramson, Monash University, Australia
     Jack Dongarra, University of Tennessee, USA
     Geoffrey Fox, Indiana University, USA
     Manish Parashar, Rutgers University, USA
     Marc Snir, Argonne National Laboratory & University of Illinois at Urbana Champaign, USA
     Xian-He Sun, Illinois Institute of Technology, USA
     Weimin Zheng, Tsinghua University, China

Program Committee
         Samer Al-Kiswany (University of British Columbia)
         Mihai Budiu (Microsoft Research)
         Kyle Chard (University of Chicago)
         Yong Chen (Texas Tech University)
         Evangelinos Constantinos (Massachusetts Institute of Technology)
         Catalin Dumitrescu (Fermi National Labs)
         Alexandru Iosup (Delft University of Technology - Netherlands)
         Florin Isaila (Universidad Carlos III de Madrid )
         Kamil Iskra (Argonne National Laboratory)
         Hui Jin (Oracle Corporation)
         Daniel Katz (University of Chicago)
         Zhiling Lan (Illinois Institute of Technology)
         Mike Lang (Los Alamos National Laboratory)
         Christopher Moretti (Princeton University)
         Bogdan Nicolae (IBM Research)
         David O'Hallaron (Carnegie Mellon University & Intel Laboratory)
         Marlon Pierce (Indiana University)
         Judy Qiu (Indiana University)
         Wei Tang (Argonne National Laboratory)
         Edward Walker (Whitworth University)
         Matthew Woitaszek (Walmart Labs)
         Ken Yocum (University of California at San Diego)
         Zhifeng Yun (Louisiana State University)
         Zhao Zhang (University of Chicago)
         Ziming Zheng (Illinois Institute of Technology)


-- 
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor, Illinois Institute of Technology (IIT)
Guest Research Faculty, Argonne National Laboratory (ANL)
=================================================================
Data-Intensive Distributed Systems Laboratory, CS/IIT
Distributed Systems Laboratory, MCS/ANL
=================================================================
Editor: IEEE TCC, Springer JoCCASA
Chair:  IEEE/ACM MTAGS, ACM ScienceCloud, IEEE/ACM DataCloud
=================================================================
Cel:      1-847-722-0876
Office:   1-312-567-5704
Email:    iraicu at cs.iit.edu
Web:      http://www.cs.iit.edu/~iraicu/
Web:      http://datasys.cs.iit.edu/
LinkedIn: http://www.linkedin.com/in/ioanraicu
Google:   http://scholar.google.com/citations?user=jE73HYAAAAAJ
=================================================================
=================================================================



From wilde at mcs.anl.gov  Sat Jun  8 12:00:26 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Sat, 8 Jun 2013 12:00:26 -0500 (CDT)
Subject: [Swift-devel] Swift standard library
In-Reply-To: <CAMUuvirxivEMpr7Uak2N3F7qtKhAXttCRnj8unOFK4d1tuO60g@mail.gmail.com>
Message-ID: <1821389592.1530236.1370710826823.JavaMail.root@mcs.anl.gov>

was: Re: Swift team meeting today - 2PM central - Agenda

Ketan, what did people think about this? Did you discuss what Swift/T has?

How would this be driven by real needs?

What doc would be needed to make it useful and used?

- Mike

----- Original Message -----
> From: "Ketan Maheshwari" <ketancmaheshwari at gmail.com>
> To: "Justin M Wozniak" <wozniak at mcs.anl.gov>
> Cc: "Michael Wilde" <wilde at mcs.anl.gov>, "David Kelly" <davidk at ci.uchicago.edu>, "Mihael Hategan"
> <hategan at mcs.anl.gov>, "Ketan Maheshwari" <ketan at mcs.anl.gov>, "Tim Armstrong" <tga at uchicago.edu>, "Yadu Nand"
> <yadudoc1729 at gmail.com>, "Scott Krieder" <skrieder at iit.edu>, "Daniel S. Katz" <dsk at ci.uchicago.edu>, "Swift Devel"
> <swift-devel at ci.uchicago.edu>
> Sent: Saturday, June 8, 2013 10:28:52 AM
> Subject: Re: Swift team meeting today - 2PM central - Agenda
> 
> 
> Additionally, I proposed we create a small standard library of basic
> apps and ship it with Swift. This library could contain app
> declarations for simple apps such as cat, hostname, date, etc. This
> library could then be imported into a Swift script making Swift
> script look compact. Will add reusability value too.
> 
> 
> We could steadily grow this to add more app declarations, say from
> SwiftApps repo.
> 
> 
> 
> On Fri, Jun 7, 2013 at 2:39 PM, Justin M Wozniak <
> wozniak at mcs.anl.gov > wrote:
> 
> 
> On 06/07/2013 12:52 PM, Michael Wilde wrote:
> 
> Ketan is working on two power grid apps and had some minor questions
> about Coasters for Mihael.
> 
> Yadu is working on the multi-site test suite which is itself
> implemented as a Swift script.
> 
> Scott is looking at CUDA BLAS, OOPS, and mdproxy apps for GPU.
> 
> Justin added some minor new syntax in Swift/T. Made significant
> additions to the Swift/T web materials. Provided Scott with an
> eigenvalue code based on BLAS compatible with our model for Swift/T
> and the GPU. Lots of conversation with the RDCEP and APS app cases.
> For RDCEP, now have a better sense of how to validate and compare
> the Swift/T implementation to the previous pure Fortran/MPI app. For
> APS, have a better sense of overall workflow and some ideas on where
> to find concurrency to target with Swift/T. Also working with Ketan
> on Swift storage access on the cloud, focusing on the Chirp
> experiments.
> 
> Tim is working on fixing up some remaining broken Swift/T tests.
> 
> Mihael is looking at some coasters data movement timeout errors
> reported by Yadu.
> 
> 
> 
> 
> o David outline strategy for swift command rework (esp. tc wildcards)
> David is investigating pre-processing the Swift/K command line with
> Perl. Mihael suggested some improvements to automatic app
> configuration without tc.data .
> 
> 
> 
> o David: outline priority for corresponding User Guide revamp
> David will have more time shortly to work on this.
> 
> 
> 
> o David: plans to phase in swift-lang.org and SWFT email list
> David is in touch with David Forero about this.
> 
> 
> 
> o Mihael: help Yadu and David start documenting the protocol for
> coasters in prep for coaster provider staging file caching feature
> work
> Done somewhere- Mihael will send Yadu and David the link.
> 
> 
> 
> o Yadu: review your observed Swift weaknesses and discuss & file (in
> bugz) suggestions to address each
> o Justin, Tim: next steps in Coaster integration
> Yadu has the coasters C client running. Discussed a list of next
> steps with Justin, including wrapping the client in SWIG and running
> it from Swift/T. Justin provided code samples for reference.
> 
> 
> 
> o Ketan: perhaps discuss next steps for making AWS resources easy to
> use by Swift end users
> Ketan is doing this for the power grid app people.
> 
> 
> 
> 
> o All: any other topics
> 
> Please report anything notable to swift-devel
> 
> Thanks, All, and have a great weekend.
> 
> - Mike
> 
> 
> 
> 
> --
> Justin M Wozniak
> 
> 
> 
> 
> 
> --
> Ketan
> 
> 


From ketancmaheshwari at gmail.com  Sat Jun  8 15:49:34 2013
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Sat, 8 Jun 2013 15:49:34 -0500
Subject: [Swift-devel] Swift standard library
In-Reply-To: <1821389592.1530236.1370710826823.JavaMail.root@mcs.anl.gov>
References: <CAMUuvirxivEMpr7Uak2N3F7qtKhAXttCRnj8unOFK4d1tuO60g@mail.gmail.com>
	<1821389592.1530236.1370710826823.JavaMail.root@mcs.anl.gov>
Message-ID: <CAMUuviqQCWeuheS3EH8jxBfG53kPwVoWitUqJ+3DODzuFEM6tg@mail.gmail.com>

Mike, my answers below:

On Sat, Jun 8, 2013 at 12:00 PM, Michael Wilde <wilde at mcs.anl.gov> wrote:

> was: Re: Swift team meeting today - 2PM central - Agenda
>
> Ketan, what did people think about this? Did you discuss what Swift/T has?
>

Mihael opined that it makes sense for small and simple apps but worried it
could grow out of control over time as there are numerous ways in which
apps could be written currently. I know Swift/T uses import lot but we did
not discuss it in app library context.


>
> How would this be driven by real needs?
>

Start with demoing the idea by using import "applib" in Swift source so
users do not need to write app declarations in there code. They just need
to define data, loops and make app calls. 'type file;' should also be part
of applib as it is used in 99% of Swift scripts. Promote this style of
programming and gradually build app libraries for science use cases by
domain say "chemlib", "mdlib", and so on.


>
> What doc would be needed to make it useful and used?
>

We will need to identify app patterns and document them. Then try to fit as
many real app calls as possible in these patterns. We will need to document
and illustrate their usage. There will be outliers which won't fit in any
patterns and then there will be users who would want to declare there own
apps. Hopefully, users would be encouraged to put their apps in their own
libs and re-use them. We could absorb new apps into our applibs as they get
widely used.



>
> - Mike
>
> ----- Original Message -----
> > From: "Ketan Maheshwari" <ketancmaheshwari at gmail.com>
> > To: "Justin M Wozniak" <wozniak at mcs.anl.gov>
> > Cc: "Michael Wilde" <wilde at mcs.anl.gov>, "David Kelly" <
> davidk at ci.uchicago.edu>, "Mihael Hategan"
> > <hategan at mcs.anl.gov>, "Ketan Maheshwari" <ketan at mcs.anl.gov>, "Tim
> Armstrong" <tga at uchicago.edu>, "Yadu Nand"
> > <yadudoc1729 at gmail.com>, "Scott Krieder" <skrieder at iit.edu>, "Daniel S.
> Katz" <dsk at ci.uchicago.edu>, "Swift Devel"
> > <swift-devel at ci.uchicago.edu>
> > Sent: Saturday, June 8, 2013 10:28:52 AM
> > Subject: Re: Swift team meeting today - 2PM central - Agenda
> >
> >
> > Additionally, I proposed we create a small standard library of basic
> > apps and ship it with Swift. This library could contain app
> > declarations for simple apps such as cat, hostname, date, etc. This
> > library could then be imported into a Swift script making Swift
> > script look compact. Will add reusability value too.
> >
> >
> > We could steadily grow this to add more app declarations, say from
> > SwiftApps repo.
> >
> >
> >
> > On Fri, Jun 7, 2013 at 2:39 PM, Justin M Wozniak <
> > wozniak at mcs.anl.gov > wrote:
> >
> >
> > On 06/07/2013 12:52 PM, Michael Wilde wrote:
> >
> > Ketan is working on two power grid apps and had some minor questions
> > about Coasters for Mihael.
> >
> > Yadu is working on the multi-site test suite which is itself
> > implemented as a Swift script.
> >
> > Scott is looking at CUDA BLAS, OOPS, and mdproxy apps for GPU.
> >
> > Justin added some minor new syntax in Swift/T. Made significant
> > additions to the Swift/T web materials. Provided Scott with an
> > eigenvalue code based on BLAS compatible with our model for Swift/T
> > and the GPU. Lots of conversation with the RDCEP and APS app cases.
> > For RDCEP, now have a better sense of how to validate and compare
> > the Swift/T implementation to the previous pure Fortran/MPI app. For
> > APS, have a better sense of overall workflow and some ideas on where
> > to find concurrency to target with Swift/T. Also working with Ketan
> > on Swift storage access on the cloud, focusing on the Chirp
> > experiments.
> >
> > Tim is working on fixing up some remaining broken Swift/T tests.
> >
> > Mihael is looking at some coasters data movement timeout errors
> > reported by Yadu.
> >
> >
> >
> >
> > o David outline strategy for swift command rework (esp. tc wildcards)
> > David is investigating pre-processing the Swift/K command line with
> > Perl. Mihael suggested some improvements to automatic app
> > configuration without tc.data .
> >
> >
> >
> > o David: outline priority for corresponding User Guide revamp
> > David will have more time shortly to work on this.
> >
> >
> >
> > o David: plans to phase in swift-lang.org and SWFT email list
> > David is in touch with David Forero about this.
> >
> >
> >
> > o Mihael: help Yadu and David start documenting the protocol for
> > coasters in prep for coaster provider staging file caching feature
> > work
> > Done somewhere- Mihael will send Yadu and David the link.
> >
> >
> >
> > o Yadu: review your observed Swift weaknesses and discuss & file (in
> > bugz) suggestions to address each
> > o Justin, Tim: next steps in Coaster integration
> > Yadu has the coasters C client running. Discussed a list of next
> > steps with Justin, including wrapping the client in SWIG and running
> > it from Swift/T. Justin provided code samples for reference.
> >
> >
> >
> > o Ketan: perhaps discuss next steps for making AWS resources easy to
> > use by Swift end users
> > Ketan is doing this for the power grid app people.
> >
> >
> >
> >
> > o All: any other topics
> >
> > Please report anything notable to swift-devel
> >
> > Thanks, All, and have a great weekend.
> >
> > - Mike
> >
> >
> >
> >
> > --
> > Justin M Wozniak
> >
> >
> >
> >
> >
> > --
> > Ketan
> >
> >
>



-- 
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130608/8cc9bb66/attachment.html>

From wilde at mcs.anl.gov  Mon Jun 10 10:55:54 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Mon, 10 Jun 2013 10:55:54 -0500 (CDT)
Subject: [Swift-devel] Status of ssh data provider tests
Message-ID: <598050339.2011428.1370879754465.JavaMail.root@mcs.anl.gov>


Hi David,

Yadu neesd assistance with the following, for the ex-search app:

1. To run from swift.rcc to Beagle using either ssh data staging or gsiftp data staging.

(I would like to also use this for two demo/tutorials that we're developing this week for Sunday and Monday presentation.)

Can you work with Yadu to test these and verify that Yadu can replicate the results.

2. To apply cdm direct mode for the swift.rcc-to-midway part of the ex-search app.  I think we showed how to do this in the last Swift RCC tutorial. Can you help Yadu to locate that (or create from scratch if no), test it, and integrate it into the ex-search app?

Thanks,

- Mike


From wilde at mcs.anl.gov  Mon Jun 10 11:08:21 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Mon, 10 Jun 2013 11:08:21 -0500 (CDT)
Subject: [Swift-devel] Status of ssh data provider tests
In-Reply-To: <598050339.2011428.1370879754465.JavaMail.root@mcs.anl.gov>
Message-ID: <553666674.2029762.1370880501533.JavaMail.root@mcs.anl.gov>


> 2. To apply cdm direct mode for the swift.rcc-to-midway part of the
> ex-search app.  I think we showed how to do this in the last Swift
> RCC tutorial. Can you help Yadu to locate that (or create from
> scratch if no), test it, and integrate it into the ex-search app?

Correction: the 2013-04-09 tutorial did not use CDM. You'll need to find examples elsewhere or create from scratch.


From davidk at ci.uchicago.edu  Mon Jun 10 12:47:14 2013
From: davidk at ci.uchicago.edu (David Kelly)
Date: Mon, 10 Jun 2013 12:47:14 -0500 (CDT)
Subject: [Swift-devel] Status of ssh data provider tests
In-Reply-To: <598050339.2011428.1370879754465.JavaMail.root@mcs.anl.gov>
Message-ID: <800563473.3799835.1370886434314.JavaMail.root@ci.uchicago.edu>

The data staging is the more difficult of the two. There are four options I can think of to get around the coaster provider staging issue. 

a. Use the ssh provider instead of ssh-cl and set filesystem to ssh. This requires setting up an auth.defaults with a stored password or passphrase (to avoid being prompted) 
b. Use the ssh provider and the built-in gridftp. Anything that uses gridftp obviously requires the usual steps of setting up grid certificates. 
c. Use ssh-cl and built-in gsiftp together. I think option is impossible due to the way certificates currently work. ssh-cl will create it's own certificate for coaster security, so if you have X509* defined I don't think things will work. 
d. Try to get things working with wrapper staging. It's possible, but will take more time, discussion, and testing. 

Is the need to set up auth.defaults a deal breaker for this? 

----- Original Message -----

> From: "Michael Wilde" <wilde at mcs.anl.gov>
> To: "David Kelly" <davidk at ci.uchicago.edu>
> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> Sent: Monday, June 10, 2013 10:55:54 AM
> Subject: Status of ssh data provider tests

> Hi David,

> Yadu neesd assistance with the following, for the ex-search app:

> 1. To run from swift.rcc to Beagle using either ssh data staging or
> gsiftp data staging.

> (I would like to also use this for two demo/tutorials that we're
> developing this week for Sunday and Monday presentation.)

> Can you work with Yadu to test these and verify that Yadu can
> replicate the results.

> 2. To apply cdm direct mode for the swift.rcc-to-midway part of the
> ex-search app. I think we showed how to do this in the last Swift
> RCC tutorial. Can you help Yadu to locate that (or create from
> scratch if no), test it, and integrate it into the ex-search app?

> Thanks,

> - Mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130610/8e9cf60b/attachment.html>

From davidk at ci.uchicago.edu  Tue Jun 11 07:41:38 2013
From: davidk at ci.uchicago.edu (David Kelly)
Date: Tue, 11 Jun 2013 07:41:38 -0500 (CDT)
Subject: [Swift-devel] Status of ssh data provider tests
In-Reply-To: <800563473.3799835.1370886434314.JavaMail.root@ci.uchicago.edu>
Message-ID: <2065969348.3939290.1370954498867.JavaMail.root@ci.uchicago.edu>

Yadu, 

I added a siteguide entry that gives the steps for setting up a run with the ssh provider. It's at http://www.ci.uchicago.edu/swift/guides/release-0.94/siteguide/siteguide.html#_ssh. I tested this from Midway to Beagle. The only difference is that on Beagle, $HOME/.ssh/authorized_keys is read only. When you get to that step, log on to accounts.ci.uchicago.edu and add your new SSH key there. That will update your authorized_keys file (takes a few minutes). I've only tested staging with a single small text file so far. Please let me know if you run into any problems. 

David 

----- Original Message -----

> From: "David Kelly" <davidk at ci.uchicago.edu>
> To: "Michael Wilde" <wilde at mcs.anl.gov>
> Cc: "Yadu Nand B" <yadunandb at ci.uchicago.edu>, "Swift Devel"
> <swift-devel at ci.uchicago.edu>
> Sent: Monday, June 10, 2013 12:47:14 PM
> Subject: Re: [Swift-devel] Status of ssh data provider tests

> The data staging is the more difficult of the two. There are four
> options I can think of to get around the coaster provider staging
> issue.

> a. Use the ssh provider instead of ssh-cl and set filesystem to ssh.
> This requires setting up an auth.defaults with a stored password or
> passphrase (to avoid being prompted)
> b. Use the ssh provider and the built-in gridftp. Anything that uses
> gridftp obviously requires the usual steps of setting up grid
> certificates.
> c. Use ssh-cl and built-in gsiftp together. I think option is
> impossible due to the way certificates currently work. ssh-cl will
> create it's own certificate for coaster security, so if you have
> X509* defined I don't think things will work.
> d. Try to get things working with wrapper staging. It's possible, but
> will take more time, discussion, and testing.

> Is the need to set up auth.defaults a deal breaker for this?

> ----- Original Message -----

> > From: "Michael Wilde" <wilde at mcs.anl.gov>
> 
> > To: "David Kelly" <davidk at ci.uchicago.edu>
> 
> > Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> 
> > Sent: Monday, June 10, 2013 10:55:54 AM
> 
> > Subject: Status of ssh data provider tests
> 

> > Hi David,
> 

> > Yadu neesd assistance with the following, for the ex-search app:
> 

> > 1. To run from swift.rcc to Beagle using either ssh data staging or
> > gsiftp data staging.
> 

> > (I would like to also use this for two demo/tutorials that we're
> > developing this week for Sunday and Monday presentation.)
> 

> > Can you work with Yadu to test these and verify that Yadu can
> > replicate the results.
> 

> > 2. To apply cdm direct mode for the swift.rcc-to-midway part of the
> > ex-search app. I think we showed how to do this in the last Swift
> > RCC tutorial. Can you help Yadu to locate that (or create from
> > scratch if no), test it, and integrate it into the ex-search app?
> 

> > Thanks,
> 

> > - Mike
> 

> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130611/f5985843/attachment.html>

From davidk at ci.uchicago.edu  Tue Jun 11 15:37:03 2013
From: davidk at ci.uchicago.edu (David Kelly)
Date: Tue, 11 Jun 2013 15:37:03 -0500 (CDT)
Subject: [Swift-devel] sourceforge SVN changes
In-Reply-To: <1370227449.30798.4.camel@echo>
Message-ID: <696107802.4123166.1370983023309.JavaMail.root@ci.uchicago.edu>

This seems to have broken the cog email updates for the last week or so. The automated tests may also have been running old code. Things should be working again now. 

It seems like I can checkout using https fine, but need to use svn+ssh to commit or else it fails with an authentication error. Not sure if it's just me or if this a new requirement. I'll update the site with the new location. 

David 
----- Original Message -----

> From: "Mihael Hategan" <hategan at mcs.anl.gov>
> To: "Swift Devel" <swift-devel at ci.uchicago.edu>
> Sent: Sunday, June 2, 2013 9:44:09 PM
> Subject: [Swift-devel] sourceforge SVN changes

> Hi,

> Sourceforge has migrated the svn repos. If you have an existing
> checkout
> you should:

> 1. make sure you have your public key in your account:
> https://sourceforge.net/account/ssh

> 2. do a 'svn relocate "svn
> +ssh://<username>@svn.code.sf.net/p/cogkit/svn/"'

> Also, we should update our checkout instructions. I believe they
> still
> support https in theory, but when I tried "https" instead of
> "svn+ssh" I
> got an "Internal server error".

> Mihael

> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130611/f01d116a/attachment.html>

From wilde at mcs.anl.gov  Wed Jun 12 09:47:26 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 12 Jun 2013 09:47:26 -0500 (CDT)
Subject: [Swift-devel] Coaster protocol documentation and questions
In-Reply-To: <1370633665.25255.1.camel@echo>
Message-ID: <484772527.2584686.1371048446870.JavaMail.root@mcs.anl.gov>

Following up from last Friday's developer meeting:

> > o Mihael: help Yadu and David start documenting the protocol for
> > coasters in prep for coaster provider staging file caching feature
> > work
>
> http://wiki.cogkit.org/wiki/Coaster_Protocol
 
This document is an excellent start. I'd forgotten that you created it - very nice.

One thing I can't see from reading the document - in part due to the symmetry of the protocol - is what parties the messages flow between in what order for various operations.

It seems that to fully understand the protocol, we should also document, using sequence charts, the message flow between the 3 parties involved for operations like "initialize", "run a task", "stage a file to task", "stage a file from task", "stage initial files".

The 3 parties are Swift (coaster client), coaster service, and coaster worker.

Im also curious what changes in the protocol (if anything) when the coaster service is embedded in the Swift JVM.

Can you augment the document with these things, or help Yadu and David to do so from logs?

Thanks,

- Mike



From yadudoc1729 at gmail.com  Wed Jun 12 11:35:55 2013
From: yadudoc1729 at gmail.com (Yadu Nand)
Date: Wed, 12 Jun 2013 11:35:55 -0500
Subject: [Swift-devel] Feedback loops in swift fails with deeper recursion
Message-ID: <CANa904kgeTkPU0BXU9LCCD_Ra69AoeZGM7jw3zTD4KjXwvG_Dw@mail.gmail.com>

The following recursion loop fails at depth >= 50. Swift is generating
intermediate files by appending
a string to the end of each intermediate file (in my case 5 chars). At
depth 50 this exceeds the 255
char limit on file names and swift fails with an -1 error code from
the user application which could
not create the filename swift gave it.

(tested on Swift 0.94 swift-r6414)

// Recursion loop
(file out) reiterate (file input, int cond){
      int rval = readData(input);
      //tracef("Content passes to reiterate is = %i \n", rval);
      if ( check(input, cond) == 1 ){
        out = input;
        tracef("Reached tail end. Terminating! \n");
      }else{
        //file temp = process(input);
        //out = reiterate(temp, cond);
        out = reiterate(process(input), cond);
      }
}

Is there a better way to do feedback of results from the app to
itself? Having the app rewrite the input
file with the result is IMHO a hack.

-Yadu


From wilde at mcs.anl.gov  Wed Jun 12 11:53:00 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 12 Jun 2013 11:53:00 -0500 (CDT)
Subject: [Swift-devel] Feedback loops in swift fails with deeper
 recursion
In-Reply-To: <CANa904kgeTkPU0BXU9LCCD_Ra69AoeZGM7jw3zTD4KjXwvG_Dw@mail.gmail.com>
Message-ID: <1021304333.2652430.1371055980461.JavaMail.root@mcs.anl.gov>

Yadu, I find your example hard to understand.  Can you clean it up to show what you are really trying to do here?

I dont see where cond changes to terminate the recursion.

Presumably its either for creating a test or for learning the language, but as-it its a bit confusing.

It would also help to show the full script.

If Swift's "concurrent" (default) mapper keeps extending the file name, you should map the files involved explicitly to a name that uses a short integer thats passed as an argument.

- Mike


----- Original Message -----
> From: "Yadu Nand" <yadudoc1729 at gmail.com>
> To: "swift-devel" <swift-devel at ci.uchicago.edu>
> Sent: Wednesday, June 12, 2013 11:35:55 AM
> Subject: [Swift-devel] Feedback loops in swift fails with deeper recursion
> 
> The following recursion loop fails at depth >= 50. Swift is
> generating
> intermediate files by appending
> a string to the end of each intermediate file (in my case 5 chars).
> At
> depth 50 this exceeds the 255
> char limit on file names and swift fails with an -1 error code from
> the user application which could
> not create the filename swift gave it.
> 
> (tested on Swift 0.94 swift-r6414)
> 
> // Recursion loop
> (file out) reiterate (file input, int cond){
>       int rval = readData(input);
>       //tracef("Content passes to reiterate is = %i \n", rval);
>       if ( check(input, cond) == 1 ){
>         out = input;
>         tracef("Reached tail end. Terminating! \n");
>       }else{
>         //file temp = process(input);
>         //out = reiterate(temp, cond);
>         out = reiterate(process(input), cond);
>       }
> }
> 
> Is there a better way to do feedback of results from the app to
> itself? Having the app rewrite the input
> file with the result is IMHO a hack.
> 
> -Yadu
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 


From hategan at mcs.anl.gov  Wed Jun 12 13:09:04 2013
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 12 Jun 2013 11:09:04 -0700
Subject: [Swift-devel] Coaster protocol documentation and questions
In-Reply-To: <484772527.2584686.1371048446870.JavaMail.root@mcs.anl.gov>
References: <484772527.2584686.1371048446870.JavaMail.root@mcs.anl.gov>
Message-ID: <1371060544.13025.2.camel@echo>

On Wed, 2013-06-12 at 09:47 -0500, Michael Wilde wrote:
> Following up from last Friday's developer meeting:
> 
> > > o Mihael: help Yadu and David start documenting the protocol for
> > > coasters in prep for coaster provider staging file caching feature
> > > work
> >
> > http://wiki.cogkit.org/wiki/Coaster_Protocol
>  
> This document is an excellent start. I'd forgotten that you created it - very nice.
> 
> One thing I can't see from reading the document - in part due to the
> symmetry of the protocol - is what parties the messages flow between
> in what order for various operations.
> 
> It seems that to fully understand the protocol, we should also
> document, using sequence charts, the message flow between the 3
> parties involved for operations like "initialize", "run a task",
> "stage a file to task", "stage a file from task", "stage initial
> files".

Makes sense.

> 
> The 3 parties are Swift (coaster client), coaster service, and coaster worker.
> 
> Im also curious what changes in the protocol (if anything) when the
> coaster service is embedded in the Swift JVM.

A special in-memory channel is used for communication (like a pipe), but
not much else.




From yadudoc1729 at gmail.com  Wed Jun 12 14:19:26 2013
From: yadudoc1729 at gmail.com (Yadu Nand)
Date: Wed, 12 Jun 2013 14:19:26 -0500
Subject: [Swift-devel] Feedback loops in swift fails with deeper
	recursion
In-Reply-To: <1021304333.2652430.1371055980461.JavaMail.root@mcs.anl.gov>
References: <CANa904kgeTkPU0BXU9LCCD_Ra69AoeZGM7jw3zTD4KjXwvG_Dw@mail.gmail.com>
	<1021304333.2652430.1371055980461.JavaMail.root@mcs.anl.gov>
Message-ID: <CANa904m_77jxMVAHeOw6GNBbqBQ9_PDOf-_8ktfSPYUjjH9rcw@mail.gmail.com>

Hello Mike,

Sorry about the crude example and even worse explanation.

I'm trying to implement a simple feedback pattern in swift. Here's
what I have : An application foo( ), which feeds back its output till
ome condition cond( ) returns true. cond( ) just checks if a counter
in the file has reached 100. The work( ) app, increments the counter
in the input file by 1. Thus the following pseudo swift code should go
to a 100 level deep recursion.

(file output) foo (file input) {
       if ( input meets conditions ){
             output = input; // tail end of recursion
       } else {
             // do some work on input and pass it back to foo ( )
             file temp = work (input);
             output = foo ( temp );
       }
}

>From what I understand here, the concurrent mapper is creating
temporary files by appending the filename at each level of the
recursion with some chars resulting in filenames exceeding the 255
char limit by the 50th level. I did not know at the time when this
mail was sent out that I could just tell the mapper to name the files
differently.

I could also have used an array of files with the recursion level as
index to retain the file's progress through the recursion and use the
level as an index to previous results.

Thanks,
Yadu

On Wed, Jun 12, 2013 at 11:53 AM, Michael Wilde <wilde at mcs.anl.gov> wrote:
> Yadu, I find your example hard to understand.  Can you clean it up to show what you are really trying to do here?
>
> I dont see where cond changes to terminate the recursion.
>
> Presumably its either for creating a test or for learning the language, but as-it its a bit confusing.
>
> It would also help to show the full script.
>
> If Swift's "concurrent" (default) mapper keeps extending the file name, you should map the files involved explicitly to a name that uses a short integer thats passed as an argument.
>
> - Mike
>
>
> ----- Original Message -----
>> From: "Yadu Nand" <yadudoc1729 at gmail.com>
>> To: "swift-devel" <swift-devel at ci.uchicago.edu>
>> Sent: Wednesday, June 12, 2013 11:35:55 AM
>> Subject: [Swift-devel] Feedback loops in swift fails with deeper recursion
>>
>> The following recursion loop fails at depth >= 50. Swift is
>> generating
>> intermediate files by appending
>> a string to the end of each intermediate file (in my case 5 chars).
>> At
>> depth 50 this exceeds the 255
>> char limit on file names and swift fails with an -1 error code from
>> the user application which could
>> not create the filename swift gave it.
>>
>> (tested on Swift 0.94 swift-r6414)
>>
>> // Recursion loop
>> (file out) reiterate (file input, int cond){
>>       int rval = readData(input);
>>       //tracef("Content passes to reiterate is = %i \n", rval);
>>       if ( check(input, cond) == 1 ){
>>         out = input;
>>         tracef("Reached tail end. Terminating! \n");
>>       }else{
>>         //file temp = process(input);
>>         //out = reiterate(temp, cond);
>>         out = reiterate(process(input), cond);
>>       }
>> }
>>
>> Is there a better way to do feedback of results from the app to
>> itself? Having the app rewrite the input
>> file with the result is IMHO a hack.
>>
>> -Yadu
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>



-- 
Yadu Nand B


From hategan at mcs.anl.gov  Wed Jun 12 19:07:09 2013
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 12 Jun 2013 17:07:09 -0700
Subject: [Swift-devel] Feedback loops in swift fails with deeper
 recursion
In-Reply-To: <CANa904m_77jxMVAHeOw6GNBbqBQ9_PDOf-_8ktfSPYUjjH9rcw@mail.gmail.com>
References: <CANa904kgeTkPU0BXU9LCCD_Ra69AoeZGM7jw3zTD4KjXwvG_Dw@mail.gmail.com>
	<1021304333.2652430.1371055980461.JavaMail.root@mcs.anl.gov>
	<CANa904m_77jxMVAHeOw6GNBbqBQ9_PDOf-_8ktfSPYUjjH9rcw@mail.gmail.com>
Message-ID: <1371082029.15941.1.camel@echo>

Yep. I can see why this is happening.

Please put this in bugzilla.

Mihael

On Wed, 2013-06-12 at 14:19 -0500, Yadu Nand wrote:
> Hello Mike,
> 
> Sorry about the crude example and even worse explanation.
> 
> I'm trying to implement a simple feedback pattern in swift. Here's
> what I have : An application foo( ), which feeds back its output till
> ome condition cond( ) returns true. cond( ) just checks if a counter
> in the file has reached 100. The work( ) app, increments the counter
> in the input file by 1. Thus the following pseudo swift code should go
> to a 100 level deep recursion.
> 
> (file output) foo (file input) {
>        if ( input meets conditions ){
>              output = input; // tail end of recursion
>        } else {
>              // do some work on input and pass it back to foo ( )
>              file temp = work (input);
>              output = foo ( temp );
>        }
> }
> 
> From what I understand here, the concurrent mapper is creating
> temporary files by appending the filename at each level of the
> recursion with some chars resulting in filenames exceeding the 255
> char limit by the 50th level. I did not know at the time when this
> mail was sent out that I could just tell the mapper to name the files
> differently.
> 
> I could also have used an array of files with the recursion level as
> index to retain the file's progress through the recursion and use the
> level as an index to previous results.
> 
> Thanks,
> Yadu
> 
> On Wed, Jun 12, 2013 at 11:53 AM, Michael Wilde <wilde at mcs.anl.gov> wrote:
> > Yadu, I find your example hard to understand.  Can you clean it up to show what you are really trying to do here?
> >
> > I dont see where cond changes to terminate the recursion.
> >
> > Presumably its either for creating a test or for learning the language, but as-it its a bit confusing.
> >
> > It would also help to show the full script.
> >
> > If Swift's "concurrent" (default) mapper keeps extending the file name, you should map the files involved explicitly to a name that uses a short integer thats passed as an argument.
> >
> > - Mike
> >
> >
> > ----- Original Message -----
> >> From: "Yadu Nand" <yadudoc1729 at gmail.com>
> >> To: "swift-devel" <swift-devel at ci.uchicago.edu>
> >> Sent: Wednesday, June 12, 2013 11:35:55 AM
> >> Subject: [Swift-devel] Feedback loops in swift fails with deeper recursion
> >>
> >> The following recursion loop fails at depth >= 50. Swift is
> >> generating
> >> intermediate files by appending
> >> a string to the end of each intermediate file (in my case 5 chars).
> >> At
> >> depth 50 this exceeds the 255
> >> char limit on file names and swift fails with an -1 error code from
> >> the user application which could
> >> not create the filename swift gave it.
> >>
> >> (tested on Swift 0.94 swift-r6414)
> >>
> >> // Recursion loop
> >> (file out) reiterate (file input, int cond){
> >>       int rval = readData(input);
> >>       //tracef("Content passes to reiterate is = %i \n", rval);
> >>       if ( check(input, cond) == 1 ){
> >>         out = input;
> >>         tracef("Reached tail end. Terminating! \n");
> >>       }else{
> >>         //file temp = process(input);
> >>         //out = reiterate(temp, cond);
> >>         out = reiterate(process(input), cond);
> >>       }
> >> }
> >>
> >> Is there a better way to do feedback of results from the app to
> >> itself? Having the app rewrite the input
> >> file with the result is IMHO a hack.
> >>
> >> -Yadu
> >> _______________________________________________
> >> Swift-devel mailing list
> >> Swift-devel at ci.uchicago.edu
> >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >>
> 
> 
> 




From wilde at mcs.anl.gov  Thu Jun 13 18:56:09 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 13 Jun 2013 18:56:09 -0500 (CDT)
Subject: [Swift-devel] sourceforge SVN changes
In-Reply-To: <1370227449.30798.4.camel@echo>
Message-ID: <1069254459.3187631.1371167769750.JavaMail.root@mcs.anl.gov>



This worked for me, per Mihael:
 svn co https:://svn.code.sf.net/p/cogkit/svn/branches/4.1.10/src/cog/

David, can you update the 0.94 source download instructions?

Thanks,

- Mike


----- Original Message -----
> From: "Mihael Hategan" <hategan at mcs.anl.gov>
> To: "Swift Devel" <swift-devel at ci.uchicago.edu>
> Sent: Sunday, June 2, 2013 9:44:09 PM
> Subject: [Swift-devel] sourceforge SVN changes
> 
> Hi,
> 
> Sourceforge has migrated the svn repos. If you have an existing
> checkout
> you should:
> 
> 1. make sure you have your public key in your account:
> https://sourceforge.net/account/ssh
> 
> 2. do a 'svn relocate "svn
> +ssh://<username>@svn.code.sf.net/p/cogkit/svn/"'
> 
> Also, we should update our checkout instructions. I believe they
> still
> support https in theory, but when I tried "https" instead of
> "svn+ssh" I
> got an "Internal server error".
> 
> Mihael
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 


From wilde at mcs.anl.gov  Fri Jun 14 08:19:19 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Fri, 14 Jun 2013 08:19:19 -0500 (CDT)
Subject: [Swift-devel] coaster transfer performance
In-Reply-To: <1371181638.4197.2.camel@echo>
Message-ID: <1151969475.3225175.1371215959089.JavaMail.root@mcs.anl.gov>


Im moving this discussion to swift-devel.

Mihael's fix to the timeout problem works very well - that problem has not recurred so far as I can tell.

We then discussed performance improvements for coaster provider staging.

Mihael suggested this:

"there should be a commented-out line in GSSChannel that says gssContext.requestConf(false); (line 85)
uncomment that and re-compile, it will disable encryption"

...and observed this:

> Disabling encryption makes the stage-ins go from about 6 MB/s to 70-90 MB/s.
> 
> However, stageouts are still slow (about 4 MB/s).

I tried that, and it speeds things up a great deal. (No measurements from the fMRI demo case yet; measurements from Mihael below.)

I also tried increasing the transfer buffer size. The current buffer is 32KB.  I tried 16X (512KB) and 4X (128KB).

At 512KB, staging-in to the app goes "very fast" (again, no measures yet) but output hangs after about 25 of 100 transfers.  At 128KB, it hasn't hung yet, and goes very fast.  I too am seeing slower stage-out times from the app back to swift than stage=in times.

I'm testing on midway's shared fs right now (GPFS?) and will test on hard disk next.

But with these two fixes, things are working with very nice reliability and performance in tests so far.

The mods for these changes are below, followed by Mihael's report on testing w/o encryption.

Yadu, please integrate your coaster-provider-staging tests into the test suite and test across a range of file sizes, durations, and endpoints. Im happy to discuss this, on this list and/or in person.

Thanks,

- Mike

mid$ cat mods
modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/buffers/Buffers.java
modules/provider-coaster/resources/worker.pl
modules/karajan/src/org/globus/cog/karajan/workflow/service/channels/GSSChannel.java
mid$ svn diff
Index: modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/buffers/Buffers.java
===================================================================
--- modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/buffers/Buffers.java      (revision 3672)
+++ modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/buffers/Buffers.java      (working copy)
@@ -24,7 +24,7 @@
 public class Buffers extends Thread {
     public static final Logger logger = Logger.getLogger(Buffers.class);
 
-    public static final int ENTRY_SIZE = 32768;
+  public static final int ENTRY_SIZE = 32768 * 4;
     public static final int ENTRIES_PER_STREAM = 8;
     public static final int MAX_ENTRIES = 1024; // 32 MB
     public static final int PERFORMANCE_LOGGING_INTERVAL = 10000;
Index: modules/provider-coaster/resources/worker.pl
===================================================================
--- modules/provider-coaster/resources/worker.pl        (revision 3672)
+++ modules/provider-coaster/resources/worker.pl        (working copy)
@@ -134,7 +134,7 @@
 my $JOB_COUNT = 0;
 
 use constant BUFSZ => 2048;
-use constant IOBUFSZ => 32768;
+use constant IOBUFSZ => 32768 * 4;
 use constant IOBLOCKSZ => 8;
 
 # If true, enable a profile result that is written to the log
Index: modules/karajan/src/org/globus/cog/karajan/workflow/service/channels/GSSChannel.java
===================================================================
--- modules/karajan/src/org/globus/cog/karajan/workflow/service/channels/GSSChannel.java        (revision 3672)
+++ modules/karajan/src/org/globus/cog/karajan/workflow/service/channels/GSSChannel.java        (working copy)
@@ -82,7 +82,7 @@
 
                                gssContext.requestAnonymity(false);
                                gssContext.requestCredDeleg(false);
-                               //gssContext.requestConf(false);
+                               gssContext.requestConf(false); // Uncommented to disable encryption
                                gssContext.setOption(GSSConstants.GSS_MODE, GSIConstants.MODE_SSL);
                                gssContext.setOption(GSSConstants.DELEGATION_TYPE,
                                                GSIConstants.DELEGATION_TYPE_LIMITED);
mid$ 



----- Original Message -----
> From: "Mihael Hategan" <hategan at mcs.anl.gov>
> To: "Michael Wilde" <wilde at mcs.anl.gov>
> Sent: Thursday, June 13, 2013 10:47:18 PM
> Subject: coaster transfer performance
> 
> Hi,
> 
> This is what I'm seeing. Disabling encryption makes the stage-ins go
> from about 6 MB/s to 70-90 MB/s.
> 
> However, stageouts are still slow (about 4 MB/s). It turns out that
> this
> is not due to the shared FS. I disabled writing to disk completely,
> and
> the performance is still around 6 MB/s.
> 
> I'll need to find out why, but probably not tonight.
> 
> Mihael
> 
> 


From hategan at mcs.anl.gov  Fri Jun 14 11:58:33 2013
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 14 Jun 2013 09:58:33 -0700
Subject: [Swift-devel] coaster transfer performance
In-Reply-To: <1151969475.3225175.1371215959089.JavaMail.root@mcs.anl.gov>
References: <1151969475.3225175.1371215959089.JavaMail.root@mcs.anl.gov>
Message-ID: <1371229113.9637.1.camel@echo>

I would like to point out that I have not seen a difference on stage-out
performance when adjusting the buffer size (on the worker side). I have
*not* tried a similar experiment for the stage-in buffer size, and I
believe that before we increase that permanently, we should see if it
does indeed make a difference.

Mihael

On Fri, 2013-06-14 at 08:19 -0500, Michael Wilde wrote:
> Im moving this discussion to swift-devel.
> 
> Mihael's fix to the timeout problem works very well - that problem has not recurred so far as I can tell.
> 
> We then discussed performance improvements for coaster provider staging.
> 
> Mihael suggested this:
> 
> "there should be a commented-out line in GSSChannel that says gssContext.requestConf(false); (line 85)
> uncomment that and re-compile, it will disable encryption"
> 
> ...and observed this:
> 
> > Disabling encryption makes the stage-ins go from about 6 MB/s to 70-90 MB/s.
> > 
> > However, stageouts are still slow (about 4 MB/s).
> 
> I tried that, and it speeds things up a great deal. (No measurements from the fMRI demo case yet; measurements from Mihael below.)
> 
> I also tried increasing the transfer buffer size. The current buffer is 32KB.  I tried 16X (512KB) and 4X (128KB).
> 
> At 512KB, staging-in to the app goes "very fast" (again, no measures yet) but output hangs after about 25 of 100 transfers.  At 128KB, it hasn't hung yet, and goes very fast.  I too am seeing slower stage-out times from the app back to swift than stage=in times.
> 
> I'm testing on midway's shared fs right now (GPFS?) and will test on hard disk next.
> 
> But with these two fixes, things are working with very nice reliability and performance in tests so far.
> 
> The mods for these changes are below, followed by Mihael's report on testing w/o encryption.
> 
> Yadu, please integrate your coaster-provider-staging tests into the test suite and test across a range of file sizes, durations, and endpoints. Im happy to discuss this, on this list and/or in person.
> 
> Thanks,
> 
> - Mike
> 
> mid$ cat mods
> modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/buffers/Buffers.java
> modules/provider-coaster/resources/worker.pl
> modules/karajan/src/org/globus/cog/karajan/workflow/service/channels/GSSChannel.java
> mid$ svn diff
> Index: modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/buffers/Buffers.java
> ===================================================================
> --- modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/buffers/Buffers.java      (revision 3672)
> +++ modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/buffers/Buffers.java      (working copy)
> @@ -24,7 +24,7 @@
>  public class Buffers extends Thread {
>      public static final Logger logger = Logger.getLogger(Buffers.class);
>  
> -    public static final int ENTRY_SIZE = 32768;
> +  public static final int ENTRY_SIZE = 32768 * 4;
>      public static final int ENTRIES_PER_STREAM = 8;
>      public static final int MAX_ENTRIES = 1024; // 32 MB
>      public static final int PERFORMANCE_LOGGING_INTERVAL = 10000;
> Index: modules/provider-coaster/resources/worker.pl
> ===================================================================
> --- modules/provider-coaster/resources/worker.pl        (revision 3672)
> +++ modules/provider-coaster/resources/worker.pl        (working copy)
> @@ -134,7 +134,7 @@
>  my $JOB_COUNT = 0;
>  
>  use constant BUFSZ => 2048;
> -use constant IOBUFSZ => 32768;
> +use constant IOBUFSZ => 32768 * 4;
>  use constant IOBLOCKSZ => 8;
>  
>  # If true, enable a profile result that is written to the log
> Index: modules/karajan/src/org/globus/cog/karajan/workflow/service/channels/GSSChannel.java
> ===================================================================
> --- modules/karajan/src/org/globus/cog/karajan/workflow/service/channels/GSSChannel.java        (revision 3672)
> +++ modules/karajan/src/org/globus/cog/karajan/workflow/service/channels/GSSChannel.java        (working copy)
> @@ -82,7 +82,7 @@
>  
>                                 gssContext.requestAnonymity(false);
>                                 gssContext.requestCredDeleg(false);
> -                               //gssContext.requestConf(false);
> +                               gssContext.requestConf(false); // Uncommented to disable encryption
>                                 gssContext.setOption(GSSConstants.GSS_MODE, GSIConstants.MODE_SSL);
>                                 gssContext.setOption(GSSConstants.DELEGATION_TYPE,
>                                                 GSIConstants.DELEGATION_TYPE_LIMITED);
> mid$ 
> 
> 
> 
> ----- Original Message -----
> > From: "Mihael Hategan" <hategan at mcs.anl.gov>
> > To: "Michael Wilde" <wilde at mcs.anl.gov>
> > Sent: Thursday, June 13, 2013 10:47:18 PM
> > Subject: coaster transfer performance
> > 
> > Hi,
> > 
> > This is what I'm seeing. Disabling encryption makes the stage-ins go
> > from about 6 MB/s to 70-90 MB/s.
> > 
> > However, stageouts are still slow (about 4 MB/s). It turns out that
> > this
> > is not due to the shared FS. I disabled writing to disk completely,
> > and
> > the performance is still around 6 MB/s.
> > 
> > I'll need to find out why, but probably not tonight.
> > 
> > Mihael
> > 
> > 




From wilde at mcs.anl.gov  Fri Jun 14 13:29:54 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Fri, 14 Jun 2013 13:29:54 -0500 (CDT)
Subject: [Swift-devel] coaster transfer performance
In-Reply-To: <1371229113.9637.1.camel@echo>
Message-ID: <255121992.3510902.1371234594546.JavaMail.root@mcs.anl.gov>

I agree about testing; note though that you need to test on a shared filesystem under load before you will see where the bigger buffers make a difference.  For example, during times when "ls" on /home or /project (on midway) or /lustre (on beagle) run very slow.  Thats when reducing the number of trips of the file server makes the most difference.

- Mike


----- Original Message -----
> From: "Mihael Hategan" <hategan at mcs.anl.gov>
> To: "Michael Wilde" <wilde at mcs.anl.gov>
> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> Sent: Friday, June 14, 2013 11:58:33 AM
> Subject: Re: coaster transfer performance
> 
> I would like to point out that I have not seen a difference on
> stage-out
> performance when adjusting the buffer size (on the worker side). I
> have
> *not* tried a similar experiment for the stage-in buffer size, and I
> believe that before we increase that permanently, we should see if it
> does indeed make a difference.
> 
> Mihael
> 
> On Fri, 2013-06-14 at 08:19 -0500, Michael Wilde wrote:
> > Im moving this discussion to swift-devel.
> > 
> > Mihael's fix to the timeout problem works very well - that problem
> > has not recurred so far as I can tell.
> > 
> > We then discussed performance improvements for coaster provider
> > staging.
> > 
> > Mihael suggested this:
> > 
> > "there should be a commented-out line in GSSChannel that says
> > gssContext.requestConf(false); (line 85)
> > uncomment that and re-compile, it will disable encryption"
> > 
> > ...and observed this:
> > 
> > > Disabling encryption makes the stage-ins go from about 6 MB/s to
> > > 70-90 MB/s.
> > > 
> > > However, stageouts are still slow (about 4 MB/s).
> > 
> > I tried that, and it speeds things up a great deal. (No
> > measurements from the fMRI demo case yet; measurements from Mihael
> > below.)
> > 
> > I also tried increasing the transfer buffer size. The current
> > buffer is 32KB.  I tried 16X (512KB) and 4X (128KB).
> > 
> > At 512KB, staging-in to the app goes "very fast" (again, no
> > measures yet) but output hangs after about 25 of 100 transfers.
> >  At 128KB, it hasn't hung yet, and goes very fast.  I too am
> > seeing slower stage-out times from the app back to swift than
> > stage=in times.
> > 
> > I'm testing on midway's shared fs right now (GPFS?) and will test
> > on hard disk next.
> > 
> > But with these two fixes, things are working with very nice
> > reliability and performance in tests so far.
> > 
> > The mods for these changes are below, followed by Mihael's report
> > on testing w/o encryption.
> > 
> > Yadu, please integrate your coaster-provider-staging tests into the
> > test suite and test across a range of file sizes, durations, and
> > endpoints. Im happy to discuss this, on this list and/or in
> > person.
> > 
> > Thanks,
> > 
> > - Mike
> > 
> > mid$ cat mods
> > modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/buffers/Buffers.java
> > modules/provider-coaster/resources/worker.pl
> > modules/karajan/src/org/globus/cog/karajan/workflow/service/channels/GSSChannel.java
> > mid$ svn diff
> > Index:
> > modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/buffers/Buffers.java
> > ===================================================================
> > ---
> > modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/buffers/Buffers.java
> >      (revision 3672)
> > +++
> > modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/buffers/Buffers.java
> >      (working copy)
> > @@ -24,7 +24,7 @@
> >  public class Buffers extends Thread {
> >      public static final Logger logger =
> >      Logger.getLogger(Buffers.class);
> >  
> > -    public static final int ENTRY_SIZE = 32768;
> > +  public static final int ENTRY_SIZE = 32768 * 4;
> >      public static final int ENTRIES_PER_STREAM = 8;
> >      public static final int MAX_ENTRIES = 1024; // 32 MB
> >      public static final int PERFORMANCE_LOGGING_INTERVAL = 10000;
> > Index: modules/provider-coaster/resources/worker.pl
> > ===================================================================
> > --- modules/provider-coaster/resources/worker.pl        (revision
> > 3672)
> > +++ modules/provider-coaster/resources/worker.pl        (working
> > copy)
> > @@ -134,7 +134,7 @@
> >  my $JOB_COUNT = 0;
> >  
> >  use constant BUFSZ => 2048;
> > -use constant IOBUFSZ => 32768;
> > +use constant IOBUFSZ => 32768 * 4;
> >  use constant IOBLOCKSZ => 8;
> >  
> >  # If true, enable a profile result that is written to the log
> > Index:
> > modules/karajan/src/org/globus/cog/karajan/workflow/service/channels/GSSChannel.java
> > ===================================================================
> > ---
> > modules/karajan/src/org/globus/cog/karajan/workflow/service/channels/GSSChannel.java
> >        (revision 3672)
> > +++
> > modules/karajan/src/org/globus/cog/karajan/workflow/service/channels/GSSChannel.java
> >        (working copy)
> > @@ -82,7 +82,7 @@
> >  
> >                                 gssContext.requestAnonymity(false);
> >                                 gssContext.requestCredDeleg(false);
> > -                               //gssContext.requestConf(false);
> > +                               gssContext.requestConf(false); //
> > Uncommented to disable encryption
> >                                 gssContext.setOption(GSSConstants.GSS_MODE,
> >                                 GSIConstants.MODE_SSL);
> >                                 gssContext.setOption(GSSConstants.DELEGATION_TYPE,
> >                                                 GSIConstants.DELEGATION_TYPE_LIMITED);
> > mid$
> > 
> > 
> > 
> > ----- Original Message -----
> > > From: "Mihael Hategan" <hategan at mcs.anl.gov>
> > > To: "Michael Wilde" <wilde at mcs.anl.gov>
> > > Sent: Thursday, June 13, 2013 10:47:18 PM
> > > Subject: coaster transfer performance
> > > 
> > > Hi,
> > > 
> > > This is what I'm seeing. Disabling encryption makes the stage-ins
> > > go
> > > from about 6 MB/s to 70-90 MB/s.
> > > 
> > > However, stageouts are still slow (about 4 MB/s). It turns out
> > > that
> > > this
> > > is not due to the shared FS. I disabled writing to disk
> > > completely,
> > > and
> > > the performance is still around 6 MB/s.
> > > 
> > > I'll need to find out why, but probably not tonight.
> > > 
> > > Mihael
> > > 
> > > 
> 
> 
> 


From wilde at mcs.anl.gov  Fri Jun 14 14:57:23 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Fri, 14 Jun 2013 14:57:23 -0500 (CDT)
Subject: [Swift-devel] Please help: error "
Message-ID: <33778315.3538818.1371239843456.JavaMail.root@mcs.anl.gov>


Hi All,

Can anyone help me find the cause of the error below: "Failed to configure log file name"

Ive never seen this before.

Im getting it on midway: swift.rcc.uchicago.edu, dir /home/wilde/Projects/OHBM2013/svn:

swift$ swift -config swift.properties -tc.file apps.midway genatlas.swift -d=data_2
Swift 0.94 swift-r6549 cog-r3672 (cog modified locally)

RunID: 20130614-1954-66k4nvfe
Progress:  time: Fri, 14 Jun 2013 19:55:01 +0000
Failed to configure log file name

Thanks!

- Mike


From hategan at mcs.anl.gov  Fri Jun 14 16:28:56 2013
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 14 Jun 2013 14:28:56 -0700
Subject: [Swift-devel] Please help: error "
In-Reply-To: <33778315.3538818.1371239843456.JavaMail.root@mcs.anl.gov>
References: <33778315.3538818.1371239843456.JavaMail.root@mcs.anl.gov>
Message-ID: <1371245336.12548.8.camel@echo>

Probably a custom log4j in your classpath that does not have a file
appender.

Is there a stack trace?

Mihael

On Fri, 2013-06-14 at 14:57 -0500, Michael Wilde wrote:
> Hi All,
> 
> Can anyone help me find the cause of the error below: "Failed to configure log file name"
> 
> Ive never seen this before.
> 
> Im getting it on midway: swift.rcc.uchicago.edu, dir /home/wilde/Projects/OHBM2013/svn:
> 
> swift$ swift -config swift.properties -tc.file apps.midway genatlas.swift -d=data_2
> Swift 0.94 swift-r6549 cog-r3672 (cog modified locally)
> 
> RunID: 20130614-1954-66k4nvfe
> Progress:  time: Fri, 14 Jun 2013 19:55:01 +0000
> Failed to configure log file name
> 
> Thanks!
> 
> - Mike
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel




From davidk at ci.uchicago.edu  Fri Jun 14 16:36:10 2013
From: davidk at ci.uchicago.edu (David Kelly)
Date: Fri, 14 Jun 2013 16:36:10 -0500 (CDT)
Subject: [Swift-devel] Please help: error "
In-Reply-To: <1371245336.12548.8.camel@echo>
Message-ID: <1112793206.5211856.1371245770058.JavaMail.root@ci.uchicago.edu>

I think this is related to some changes in CoasterService.java last week. A version of Swift from a few weeks ago with the same scripts/setup doesn't produce this message. Swift seems to continue running after this, and I don't see a stack trace. 

----- Original Message -----

> From: "Mihael Hategan" <hategan at mcs.anl.gov>
> To: "Michael Wilde" <wilde at mcs.anl.gov>
> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> Sent: Friday, June 14, 2013 4:28:56 PM
> Subject: Re: [Swift-devel] Please help: error "

> Probably a custom log4j in your classpath that does not have a file
> appender.

> Is there a stack trace?

> Mihael

> On Fri, 2013-06-14 at 14:57 -0500, Michael Wilde wrote:
> > Hi All,
> >
> > Can anyone help me find the cause of the error below: "Failed to
> > configure log file name"
> >
> > Ive never seen this before.
> >
> > Im getting it on midway: swift.rcc.uchicago.edu, dir
> > /home/wilde/Projects/OHBM2013/svn:
> >
> > swift$ swift -config swift.properties -tc.file apps.midway
> > genatlas.swift -d=data_2
> > Swift 0.94 swift-r6549 cog-r3672 (cog modified locally)
> >
> > RunID: 20130614-1954-66k4nvfe
> > Progress: time: Fri, 14 Jun 2013 19:55:01 +0000
> > Failed to configure log file name
> >
> > Thanks!
> >
> > - Mike
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130614/090f3e85/attachment.html>

From hategan at mcs.anl.gov  Sat Jun 15 03:47:04 2013
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sat, 15 Jun 2013 01:47:04 -0700
Subject: [Swift-devel] Please help: error "
In-Reply-To: <1112793206.5211856.1371245770058.JavaMail.root@ci.uchicago.edu>
References: <1112793206.5211856.1371245770058.JavaMail.root@ci.uchicago.edu>
Message-ID: <1371286024.13800.4.camel@echo>

The change you mention was meant to re-name the coaster log file to
something containing the date rather than just coasters.log.

It did just occur to me that this might happen when using coaster
local:x. Is that what's happening? In other words, what's in sites.xml
for the run that prints this error?

Mihael

PS: the run should just continue and there should be no stack trace (as
David points out).

On Fri, 2013-06-14 at 16:36 -0500, David Kelly wrote:
> I think this is related to some changes in CoasterService.java last
> week. A version of Swift from a few weeks ago with the same
> scripts/setup doesn't produce this message. Swift seems to continue
> running after this, and I don't see a stack trace. 
> 
> ----- Original Message -----
> 
> > From: "Mihael Hategan" <hategan at mcs.anl.gov>
> > To: "Michael Wilde" <wilde at mcs.anl.gov>
> > Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> > Sent: Friday, June 14, 2013 4:28:56 PM
> > Subject: Re: [Swift-devel] Please help: error "
> 
> > Probably a custom log4j in your classpath that does not have a file
> > appender.
> 
> > Is there a stack trace?
> 
> > Mihael
> 
> > On Fri, 2013-06-14 at 14:57 -0500, Michael Wilde wrote:
> > > Hi All,
> > >
> > > Can anyone help me find the cause of the error below: "Failed to
> > > configure log file name"
> > >
> > > Ive never seen this before.
> > >
> > > Im getting it on midway: swift.rcc.uchicago.edu, dir
> > > /home/wilde/Projects/OHBM2013/svn:
> > >
> > > swift$ swift -config swift.properties -tc.file apps.midway
> > > genatlas.swift -d=data_2
> > > Swift 0.94 swift-r6549 cog-r3672 (cog modified locally)
> > >
> > > RunID: 20130614-1954-66k4nvfe
> > > Progress: time: Fri, 14 Jun 2013 19:55:01 +0000
> > > Failed to configure log file name
> > >
> > > Thanks!
> > >
> > > - Mike
> > > _______________________________________________
> > > Swift-devel mailing list
> > > Swift-devel at ci.uchicago.edu
> > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> I think this is related to some changes in CoasterService.java last
> week. A version of Swift from a few weeks ago with the same
> scripts/setup doesn't produce this message. Swift seems to continue
> running after this, and I don't see a stack trace.
> 
> 
> ______________________________________________________________________
>         From: "Mihael Hategan" <hategan at mcs.anl.gov>
>         To: "Michael Wilde" <wilde at mcs.anl.gov>
>         Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
>         Sent: Friday, June 14, 2013 4:28:56 PM
>         Subject: Re: [Swift-devel] Please help: error "
>         
>         Probably a custom log4j in your classpath that does not have a
>         file
>         appender.
>         
>         Is there a stack trace?
>         
>         Mihael
>         
>         On Fri, 2013-06-14 at 14:57 -0500, Michael Wilde wrote:
>         > Hi All,
>         > 
>         > Can anyone help me find the cause of the error below:
>         "Failed to configure log file name"
>         > 
>         > Ive never seen this before.
>         > 
>         > Im getting it on midway: swift.rcc.uchicago.edu,
>         dir /home/wilde/Projects/OHBM2013/svn:
>         > 
>         > swift$ swift -config swift.properties -tc.file apps.midway
>         genatlas.swift -d=data_2
>         > Swift 0.94 swift-r6549 cog-r3672 (cog modified locally)
>         > 
>         > RunID: 20130614-1954-66k4nvfe
>         > Progress:  time: Fri, 14 Jun 2013 19:55:01 +0000
>         > Failed to configure log file name
>         > 
>         > Thanks!
>         > 
>         > - Mike
>         > _______________________________________________
>         > Swift-devel mailing list
>         > Swift-devel at ci.uchicago.edu
>         >
>         https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>         
>         
>         _______________________________________________
>         Swift-devel mailing list
>         Swift-devel at ci.uchicago.edu
>         https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> 




From wilde at mcs.anl.gov  Sat Jun 15 08:58:39 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Sat, 15 Jun 2013 08:58:39 -0500 (CDT)
Subject: [Swift-devel] Please help: error "
In-Reply-To: <1371286024.13800.4.camel@echo>
Message-ID: <1293380895.3615046.1371304719936.JavaMail.root@mcs.anl.gov>


I sent this help request because I thought it was causing the demo runs to fail.

When David tested, he noted that the error was *not* causing any failures.

Testing now, I too see that the demo now seems to be working.

Its possible that I was misled by some other aspect of the run(s) I did.

So we should fix this, and verify what the impact is, but for now I *think* its OK.

My command line, output, dir on midway, and sites.xml is below.

The log for this runs is at http://www.ci.uchicago.edu/~wilde/atlas01-20130615-1355-z6imink0.log

Thanks,

- Mike

swift$ swift -tc.file apps.midway -sites.file t.xml atlas01.swift
Swift 0.94 swift-r6549 cog-r3672 (cog modified locally)

RunID: 20130615-1355-z6imink0
Progress:  time: Sat, 15 Jun 2013 13:55:47 +0000
Failed to configure log file name
Progress:  time: Sat, 15 Jun 2013 13:55:52 +0000  Stage in:1  Submitted:3
Progress:  time: Sat, 15 Jun 2013 13:55:53 +0000  Stage in:3  Active:1
Final status: Sat, 15 Jun 2013 13:55:54 +0000  Finished successfully:4
swift$ pwd
/home/wilde/Projects/OHBM2013/svn-mcs
swift$ cat t.xml
<config>

  <pool handle="localhost">
    <execution provider="local"/>
    <profile namespace="karajan" key="initialScore">10000</profile>
    <filesystem provider="local"/>
    <workdirectory>/scratch/midway/{env.USER}/swiftwork</workdirectory>
    <profile namespace="swift" key="stagingMethod">file</profile>
    <profile namespace="karajan" key="jobThrottle">.03</profile> <!-- eg .11 -> 12 -->
  </pool>

  <pool handle="midway">
    <execution provider="coaster" jobmanager="local:slurm"/>

    <!-- Set partition and account here: -->
    <profile namespace="globus" key="queue">westmere</profile>
    <profile namespace="globus" key="ppn">12</profile>
    <!-- <profile namespace="globus" key="project">pi-wilde</profile> -->

    <!-- Set number of jobs and nodes per job here: -->
    <profile namespace="globus" key="slots">1</profile>
    <profile namespace="globus" key="maxnodes">1</profile>
    <profile namespace="globus" key="nodegranularity">1</profile>
    <profile namespace="globus" key="jobsPerNode">12</profile> <!-- apps per node! -->
    <profile namespace="karajan" key="jobThrottle">.11</profile> <!-- eg .11 -> 12 -->

    <!-- Set estimated app time (maxwalltime) and requested job time (maxtime) here: -->
    <profile namespace="globus" key="maxWalltime">00:15:00</profile>
    <profile namespace="globus" key="maxtime">1800</profile>  <!-- in seconds! -->

    <!-- Set data staging model and work dir here: -->
    <filesystem provider="local"/>
    <workdirectory>/scratch/midway/{env.USER}/swiftwork</workdirectory>

    <!-- Typically leave these constant: -->
    <profile namespace="globus" key="slurm.exclusive">false</profile>
    <profile namespace="globus" key="highOverAllocation">100</profile>
    <profile namespace="globus" key="lowOverAllocation">100</profile>
    <profile namespace="karajan" key="initialScore">10000</profile>
  </pool>

</config>
swift$ 

----- Original Message -----
> From: "Mihael Hategan" <hategan at mcs.anl.gov>
> To: "David Kelly" <davidk at ci.uchicago.edu>
> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>, "Michael Wilde" <wilde at mcs.anl.gov>
> Sent: Saturday, June 15, 2013 3:47:04 AM
> Subject: Re: [Swift-devel] Please help: error "
> 
> The change you mention was meant to re-name the coaster log file to
> something containing the date rather than just coasters.log.
> 
> It did just occur to me that this might happen when using coaster
> local:x. Is that what's happening? In other words, what's in
> sites.xml
> for the run that prints this error?
> 
> Mihael
> 
> PS: the run should just continue and there should be no stack trace
> (as
> David points out).
> 
> On Fri, 2013-06-14 at 16:36 -0500, David Kelly wrote:
> > I think this is related to some changes in CoasterService.java last
> > week. A version of Swift from a few weeks ago with the same
> > scripts/setup doesn't produce this message. Swift seems to continue
> > running after this, and I don't see a stack trace.
> > 
> > ----- Original Message -----
> > 
> > > From: "Mihael Hategan" <hategan at mcs.anl.gov>
> > > To: "Michael Wilde" <wilde at mcs.anl.gov>
> > > Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> > > Sent: Friday, June 14, 2013 4:28:56 PM
> > > Subject: Re: [Swift-devel] Please help: error "
> > 
> > > Probably a custom log4j in your classpath that does not have a
> > > file
> > > appender.
> > 
> > > Is there a stack trace?
> > 
> > > Mihael
> > 
> > > On Fri, 2013-06-14 at 14:57 -0500, Michael Wilde wrote:
> > > > Hi All,
> > > >
> > > > Can anyone help me find the cause of the error below: "Failed
> > > > to
> > > > configure log file name"
> > > >
> > > > Ive never seen this before.
> > > >
> > > > Im getting it on midway: swift.rcc.uchicago.edu, dir
> > > > /home/wilde/Projects/OHBM2013/svn:
> > > >
> > > > swift$ swift -config swift.properties -tc.file apps.midway
> > > > genatlas.swift -d=data_2
> > > > Swift 0.94 swift-r6549 cog-r3672 (cog modified locally)
> > > >
> > > > RunID: 20130614-1954-66k4nvfe
> > > > Progress: time: Fri, 14 Jun 2013 19:55:01 +0000
> > > > Failed to configure log file name
> > > >
> > > > Thanks!
> > > >
> > > > - Mike
> > > > _______________________________________________
> > > > Swift-devel mailing list
> > > > Swift-devel at ci.uchicago.edu
> > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > 
> > > _______________________________________________
> > > Swift-devel mailing list
> > > Swift-devel at ci.uchicago.edu
> > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > I think this is related to some changes in CoasterService.java last
> > week. A version of Swift from a few weeks ago with the same
> > scripts/setup doesn't produce this message. Swift seems to continue
> > running after this, and I don't see a stack trace.
> > 
> > 
> > ______________________________________________________________________
> >         From: "Mihael Hategan" <hategan at mcs.anl.gov>
> >         To: "Michael Wilde" <wilde at mcs.anl.gov>
> >         Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> >         Sent: Friday, June 14, 2013 4:28:56 PM
> >         Subject: Re: [Swift-devel] Please help: error "
> >         
> >         Probably a custom log4j in your classpath that does not
> >         have a
> >         file
> >         appender.
> >         
> >         Is there a stack trace?
> >         
> >         Mihael
> >         
> >         On Fri, 2013-06-14 at 14:57 -0500, Michael Wilde wrote:
> >         > Hi All,
> >         > 
> >         > Can anyone help me find the cause of the error below:
> >         "Failed to configure log file name"
> >         > 
> >         > Ive never seen this before.
> >         > 
> >         > Im getting it on midway: swift.rcc.uchicago.edu,
> >         dir /home/wilde/Projects/OHBM2013/svn:
> >         > 
> >         > swift$ swift -config swift.properties -tc.file
> >         > apps.midway
> >         genatlas.swift -d=data_2
> >         > Swift 0.94 swift-r6549 cog-r3672 (cog modified locally)
> >         > 
> >         > RunID: 20130614-1954-66k4nvfe
> >         > Progress:  time: Fri, 14 Jun 2013 19:55:01 +0000
> >         > Failed to configure log file name
> >         > 
> >         > Thanks!
> >         > 
> >         > - Mike
> >         > _______________________________________________
> >         > Swift-devel mailing list
> >         > Swift-devel at ci.uchicago.edu
> >         >
> >         https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >         
> >         
> >         _______________________________________________
> >         Swift-devel mailing list
> >         Swift-devel at ci.uchicago.edu
> >         https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > 
> > 
> 
> 
> 


From hategan at mcs.anl.gov  Sat Jun 15 18:49:32 2013
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sat, 15 Jun 2013 16:49:32 -0700
Subject: [Swift-devel] Please help: error "
In-Reply-To: <1293380895.3615046.1371304719936.JavaMail.root@mcs.anl.gov>
References: <1293380895.3615046.1371304719936.JavaMail.root@mcs.anl.gov>
Message-ID: <1371340172.18618.0.camel@echo>

This should now be fixed in SVN. Thanks David for the hint about the
coaster service change being at the root of the problem.

Mihael

On Sat, 2013-06-15 at 08:58 -0500, Michael Wilde wrote:
> I sent this help request because I thought it was causing the demo runs to fail.
> 
> When David tested, he noted that the error was *not* causing any failures.
> 
> Testing now, I too see that the demo now seems to be working.
> 
> Its possible that I was misled by some other aspect of the run(s) I did.
> 
> So we should fix this, and verify what the impact is, but for now I *think* its OK.
> 
> My command line, output, dir on midway, and sites.xml is below.
> 
> The log for this runs is at http://www.ci.uchicago.edu/~wilde/atlas01-20130615-1355-z6imink0.log
> 
> Thanks,
> 
> - Mike
> 
> swift$ swift -tc.file apps.midway -sites.file t.xml atlas01.swift
> Swift 0.94 swift-r6549 cog-r3672 (cog modified locally)
> 
> RunID: 20130615-1355-z6imink0
> Progress:  time: Sat, 15 Jun 2013 13:55:47 +0000
> Failed to configure log file name
> Progress:  time: Sat, 15 Jun 2013 13:55:52 +0000  Stage in:1  Submitted:3
> Progress:  time: Sat, 15 Jun 2013 13:55:53 +0000  Stage in:3  Active:1
> Final status: Sat, 15 Jun 2013 13:55:54 +0000  Finished successfully:4
> swift$ pwd
> /home/wilde/Projects/OHBM2013/svn-mcs
> swift$ cat t.xml
> <config>
> 
>   <pool handle="localhost">
>     <execution provider="local"/>
>     <profile namespace="karajan" key="initialScore">10000</profile>
>     <filesystem provider="local"/>
>     <workdirectory>/scratch/midway/{env.USER}/swiftwork</workdirectory>
>     <profile namespace="swift" key="stagingMethod">file</profile>
>     <profile namespace="karajan" key="jobThrottle">.03</profile> <!-- eg .11 -> 12 -->
>   </pool>
> 
>   <pool handle="midway">
>     <execution provider="coaster" jobmanager="local:slurm"/>
> 
>     <!-- Set partition and account here: -->
>     <profile namespace="globus" key="queue">westmere</profile>
>     <profile namespace="globus" key="ppn">12</profile>
>     <!-- <profile namespace="globus" key="project">pi-wilde</profile> -->
> 
>     <!-- Set number of jobs and nodes per job here: -->
>     <profile namespace="globus" key="slots">1</profile>
>     <profile namespace="globus" key="maxnodes">1</profile>
>     <profile namespace="globus" key="nodegranularity">1</profile>
>     <profile namespace="globus" key="jobsPerNode">12</profile> <!-- apps per node! -->
>     <profile namespace="karajan" key="jobThrottle">.11</profile> <!-- eg .11 -> 12 -->
> 
>     <!-- Set estimated app time (maxwalltime) and requested job time (maxtime) here: -->
>     <profile namespace="globus" key="maxWalltime">00:15:00</profile>
>     <profile namespace="globus" key="maxtime">1800</profile>  <!-- in seconds! -->
> 
>     <!-- Set data staging model and work dir here: -->
>     <filesystem provider="local"/>
>     <workdirectory>/scratch/midway/{env.USER}/swiftwork</workdirectory>
> 
>     <!-- Typically leave these constant: -->
>     <profile namespace="globus" key="slurm.exclusive">false</profile>
>     <profile namespace="globus" key="highOverAllocation">100</profile>
>     <profile namespace="globus" key="lowOverAllocation">100</profile>
>     <profile namespace="karajan" key="initialScore">10000</profile>
>   </pool>
> 
> </config>
> swift$ 
> 
> ----- Original Message -----
> > From: "Mihael Hategan" <hategan at mcs.anl.gov>
> > To: "David Kelly" <davidk at ci.uchicago.edu>
> > Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>, "Michael Wilde" <wilde at mcs.anl.gov>
> > Sent: Saturday, June 15, 2013 3:47:04 AM
> > Subject: Re: [Swift-devel] Please help: error "
> > 
> > The change you mention was meant to re-name the coaster log file to
> > something containing the date rather than just coasters.log.
> > 
> > It did just occur to me that this might happen when using coaster
> > local:x. Is that what's happening? In other words, what's in
> > sites.xml
> > for the run that prints this error?
> > 
> > Mihael
> > 
> > PS: the run should just continue and there should be no stack trace
> > (as
> > David points out).
> > 
> > On Fri, 2013-06-14 at 16:36 -0500, David Kelly wrote:
> > > I think this is related to some changes in CoasterService.java last
> > > week. A version of Swift from a few weeks ago with the same
> > > scripts/setup doesn't produce this message. Swift seems to continue
> > > running after this, and I don't see a stack trace.
> > > 
> > > ----- Original Message -----
> > > 
> > > > From: "Mihael Hategan" <hategan at mcs.anl.gov>
> > > > To: "Michael Wilde" <wilde at mcs.anl.gov>
> > > > Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> > > > Sent: Friday, June 14, 2013 4:28:56 PM
> > > > Subject: Re: [Swift-devel] Please help: error "
> > > 
> > > > Probably a custom log4j in your classpath that does not have a
> > > > file
> > > > appender.
> > > 
> > > > Is there a stack trace?
> > > 
> > > > Mihael
> > > 
> > > > On Fri, 2013-06-14 at 14:57 -0500, Michael Wilde wrote:
> > > > > Hi All,
> > > > >
> > > > > Can anyone help me find the cause of the error below: "Failed
> > > > > to
> > > > > configure log file name"
> > > > >
> > > > > Ive never seen this before.
> > > > >
> > > > > Im getting it on midway: swift.rcc.uchicago.edu, dir
> > > > > /home/wilde/Projects/OHBM2013/svn:
> > > > >
> > > > > swift$ swift -config swift.properties -tc.file apps.midway
> > > > > genatlas.swift -d=data_2
> > > > > Swift 0.94 swift-r6549 cog-r3672 (cog modified locally)
> > > > >
> > > > > RunID: 20130614-1954-66k4nvfe
> > > > > Progress: time: Fri, 14 Jun 2013 19:55:01 +0000
> > > > > Failed to configure log file name
> > > > >
> > > > > Thanks!
> > > > >
> > > > > - Mike
> > > > > _______________________________________________
> > > > > Swift-devel mailing list
> > > > > Swift-devel at ci.uchicago.edu
> > > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > 
> > > > _______________________________________________
> > > > Swift-devel mailing list
> > > > Swift-devel at ci.uchicago.edu
> > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > I think this is related to some changes in CoasterService.java last
> > > week. A version of Swift from a few weeks ago with the same
> > > scripts/setup doesn't produce this message. Swift seems to continue
> > > running after this, and I don't see a stack trace.
> > > 
> > > 
> > > ______________________________________________________________________
> > >         From: "Mihael Hategan" <hategan at mcs.anl.gov>
> > >         To: "Michael Wilde" <wilde at mcs.anl.gov>
> > >         Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> > >         Sent: Friday, June 14, 2013 4:28:56 PM
> > >         Subject: Re: [Swift-devel] Please help: error "
> > >         
> > >         Probably a custom log4j in your classpath that does not
> > >         have a
> > >         file
> > >         appender.
> > >         
> > >         Is there a stack trace?
> > >         
> > >         Mihael
> > >         
> > >         On Fri, 2013-06-14 at 14:57 -0500, Michael Wilde wrote:
> > >         > Hi All,
> > >         > 
> > >         > Can anyone help me find the cause of the error below:
> > >         "Failed to configure log file name"
> > >         > 
> > >         > Ive never seen this before.
> > >         > 
> > >         > Im getting it on midway: swift.rcc.uchicago.edu,
> > >         dir /home/wilde/Projects/OHBM2013/svn:
> > >         > 
> > >         > swift$ swift -config swift.properties -tc.file
> > >         > apps.midway
> > >         genatlas.swift -d=data_2
> > >         > Swift 0.94 swift-r6549 cog-r3672 (cog modified locally)
> > >         > 
> > >         > RunID: 20130614-1954-66k4nvfe
> > >         > Progress:  time: Fri, 14 Jun 2013 19:55:01 +0000
> > >         > Failed to configure log file name
> > >         > 
> > >         > Thanks!
> > >         > 
> > >         > - Mike
> > >         > _______________________________________________
> > >         > Swift-devel mailing list
> > >         > Swift-devel at ci.uchicago.edu
> > >         >
> > >         https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > >         
> > >         
> > >         _______________________________________________
> > >         Swift-devel mailing list
> > >         Swift-devel at ci.uchicago.edu
> > >         https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > 
> > > 
> > 
> > 
> > 




From wilde at mcs.anl.gov  Sat Jun 15 19:06:49 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Sat, 15 Jun 2013 19:06:49 -0500 (CDT)
Subject: [Swift-devel] Please help: error "
In-Reply-To: <1371340172.18618.0.camel@echo>
Message-ID: <2036566823.3625680.1371341209756.JavaMail.root@mcs.anl.gov>

Wonderful, that fixes it - thanks very much!

Justin, my swift on midway is now updated with this fix, and is the best Swift to use there for Sundays demos.

$ cat ./set-swift 
PATH=/project/wilde/swift/src/0.94/cog/modules/swift/dist/swift-svn/bin:$PATH

- Mike


----- Original Message -----
> From: "Mihael Hategan" <hategan at mcs.anl.gov>
> To: "Michael Wilde" <wilde at mcs.anl.gov>
> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>, "David Kelly" <davidk at ci.uchicago.edu>
> Sent: Saturday, June 15, 2013 6:49:32 PM
> Subject: Re: [Swift-devel] Please help: error "
> 
> This should now be fixed in SVN. Thanks David for the hint about the
> coaster service change being at the root of the problem.
> 
> Mihael
> 
> On Sat, 2013-06-15 at 08:58 -0500, Michael Wilde wrote:
> > I sent this help request because I thought it was causing the demo
> > runs to fail.
> > 
> > When David tested, he noted that the error was *not* causing any
> > failures.
> > 
> > Testing now, I too see that the demo now seems to be working.
> > 
> > Its possible that I was misled by some other aspect of the run(s) I
> > did.
> > 
> > So we should fix this, and verify what the impact is, but for now I
> > *think* its OK.
> > 
> > My command line, output, dir on midway, and sites.xml is below.
> > 
> > The log for this runs is at
> > http://www.ci.uchicago.edu/~wilde/atlas01-20130615-1355-z6imink0.log
> > 
> > Thanks,
> > 
> > - Mike
> > 
> > swift$ swift -tc.file apps.midway -sites.file t.xml atlas01.swift
> > Swift 0.94 swift-r6549 cog-r3672 (cog modified locally)
> > 
> > RunID: 20130615-1355-z6imink0
> > Progress:  time: Sat, 15 Jun 2013 13:55:47 +0000
> > Failed to configure log file name
> > Progress:  time: Sat, 15 Jun 2013 13:55:52 +0000  Stage in:1
> >  Submitted:3
> > Progress:  time: Sat, 15 Jun 2013 13:55:53 +0000  Stage in:3
> >  Active:1
> > Final status: Sat, 15 Jun 2013 13:55:54 +0000  Finished
> > successfully:4
> > swift$ pwd
> > /home/wilde/Projects/OHBM2013/svn-mcs
> > swift$ cat t.xml
> > <config>
> > 
> >   <pool handle="localhost">
> >     <execution provider="local"/>
> >     <profile namespace="karajan" key="initialScore">10000</profile>
> >     <filesystem provider="local"/>
> >     <workdirectory>/scratch/midway/{env.USER}/swiftwork</workdirectory>
> >     <profile namespace="swift" key="stagingMethod">file</profile>
> >     <profile namespace="karajan" key="jobThrottle">.03</profile>
> >     <!-- eg .11 -> 12 -->
> >   </pool>
> > 
> >   <pool handle="midway">
> >     <execution provider="coaster" jobmanager="local:slurm"/>
> > 
> >     <!-- Set partition and account here: -->
> >     <profile namespace="globus" key="queue">westmere</profile>
> >     <profile namespace="globus" key="ppn">12</profile>
> >     <!-- <profile namespace="globus"
> >     key="project">pi-wilde</profile> -->
> > 
> >     <!-- Set number of jobs and nodes per job here: -->
> >     <profile namespace="globus" key="slots">1</profile>
> >     <profile namespace="globus" key="maxnodes">1</profile>
> >     <profile namespace="globus" key="nodegranularity">1</profile>
> >     <profile namespace="globus" key="jobsPerNode">12</profile> <!--
> >     apps per node! -->
> >     <profile namespace="karajan" key="jobThrottle">.11</profile>
> >     <!-- eg .11 -> 12 -->
> > 
> >     <!-- Set estimated app time (maxwalltime) and requested job
> >     time (maxtime) here: -->
> >     <profile namespace="globus"
> >     key="maxWalltime">00:15:00</profile>
> >     <profile namespace="globus" key="maxtime">1800</profile>  <!--
> >     in seconds! -->
> > 
> >     <!-- Set data staging model and work dir here: -->
> >     <filesystem provider="local"/>
> >     <workdirectory>/scratch/midway/{env.USER}/swiftwork</workdirectory>
> > 
> >     <!-- Typically leave these constant: -->
> >     <profile namespace="globus"
> >     key="slurm.exclusive">false</profile>
> >     <profile namespace="globus"
> >     key="highOverAllocation">100</profile>
> >     <profile namespace="globus"
> >     key="lowOverAllocation">100</profile>
> >     <profile namespace="karajan" key="initialScore">10000</profile>
> >   </pool>
> > 
> > </config>
> > swift$
> > 
> > ----- Original Message -----
> > > From: "Mihael Hategan" <hategan at mcs.anl.gov>
> > > To: "David Kelly" <davidk at ci.uchicago.edu>
> > > Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>, "Michael Wilde"
> > > <wilde at mcs.anl.gov>
> > > Sent: Saturday, June 15, 2013 3:47:04 AM
> > > Subject: Re: [Swift-devel] Please help: error "
> > > 
> > > The change you mention was meant to re-name the coaster log file
> > > to
> > > something containing the date rather than just coasters.log.
> > > 
> > > It did just occur to me that this might happen when using coaster
> > > local:x. Is that what's happening? In other words, what's in
> > > sites.xml
> > > for the run that prints this error?
> > > 
> > > Mihael
> > > 
> > > PS: the run should just continue and there should be no stack
> > > trace
> > > (as
> > > David points out).
> > > 
> > > On Fri, 2013-06-14 at 16:36 -0500, David Kelly wrote:
> > > > I think this is related to some changes in CoasterService.java
> > > > last
> > > > week. A version of Swift from a few weeks ago with the same
> > > > scripts/setup doesn't produce this message. Swift seems to
> > > > continue
> > > > running after this, and I don't see a stack trace.
> > > > 
> > > > ----- Original Message -----
> > > > 
> > > > > From: "Mihael Hategan" <hategan at mcs.anl.gov>
> > > > > To: "Michael Wilde" <wilde at mcs.anl.gov>
> > > > > Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> > > > > Sent: Friday, June 14, 2013 4:28:56 PM
> > > > > Subject: Re: [Swift-devel] Please help: error "
> > > > 
> > > > > Probably a custom log4j in your classpath that does not have
> > > > > a
> > > > > file
> > > > > appender.
> > > > 
> > > > > Is there a stack trace?
> > > > 
> > > > > Mihael
> > > > 
> > > > > On Fri, 2013-06-14 at 14:57 -0500, Michael Wilde wrote:
> > > > > > Hi All,
> > > > > >
> > > > > > Can anyone help me find the cause of the error below:
> > > > > > "Failed
> > > > > > to
> > > > > > configure log file name"
> > > > > >
> > > > > > Ive never seen this before.
> > > > > >
> > > > > > Im getting it on midway: swift.rcc.uchicago.edu, dir
> > > > > > /home/wilde/Projects/OHBM2013/svn:
> > > > > >
> > > > > > swift$ swift -config swift.properties -tc.file apps.midway
> > > > > > genatlas.swift -d=data_2
> > > > > > Swift 0.94 swift-r6549 cog-r3672 (cog modified locally)
> > > > > >
> > > > > > RunID: 20130614-1954-66k4nvfe
> > > > > > Progress: time: Fri, 14 Jun 2013 19:55:01 +0000
> > > > > > Failed to configure log file name
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > - Mike
> > > > > > _______________________________________________
> > > > > > Swift-devel mailing list
> > > > > > Swift-devel at ci.uchicago.edu
> > > > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > > 
> > > > > _______________________________________________
> > > > > Swift-devel mailing list
> > > > > Swift-devel at ci.uchicago.edu
> > > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > > I think this is related to some changes in CoasterService.java
> > > > last
> > > > week. A version of Swift from a few weeks ago with the same
> > > > scripts/setup doesn't produce this message. Swift seems to
> > > > continue
> > > > running after this, and I don't see a stack trace.
> > > > 
> > > > 
> > > > ______________________________________________________________________
> > > >         From: "Mihael Hategan" <hategan at mcs.anl.gov>
> > > >         To: "Michael Wilde" <wilde at mcs.anl.gov>
> > > >         Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> > > >         Sent: Friday, June 14, 2013 4:28:56 PM
> > > >         Subject: Re: [Swift-devel] Please help: error "
> > > >         
> > > >         Probably a custom log4j in your classpath that does not
> > > >         have a
> > > >         file
> > > >         appender.
> > > >         
> > > >         Is there a stack trace?
> > > >         
> > > >         Mihael
> > > >         
> > > >         On Fri, 2013-06-14 at 14:57 -0500, Michael Wilde wrote:
> > > >         > Hi All,
> > > >         > 
> > > >         > Can anyone help me find the cause of the error below:
> > > >         "Failed to configure log file name"
> > > >         > 
> > > >         > Ive never seen this before.
> > > >         > 
> > > >         > Im getting it on midway: swift.rcc.uchicago.edu,
> > > >         dir /home/wilde/Projects/OHBM2013/svn:
> > > >         > 
> > > >         > swift$ swift -config swift.properties -tc.file
> > > >         > apps.midway
> > > >         genatlas.swift -d=data_2
> > > >         > Swift 0.94 swift-r6549 cog-r3672 (cog modified
> > > >         > locally)
> > > >         > 
> > > >         > RunID: 20130614-1954-66k4nvfe
> > > >         > Progress:  time: Fri, 14 Jun 2013 19:55:01 +0000
> > > >         > Failed to configure log file name
> > > >         > 
> > > >         > Thanks!
> > > >         > 
> > > >         > - Mike
> > > >         > _______________________________________________
> > > >         > Swift-devel mailing list
> > > >         > Swift-devel at ci.uchicago.edu
> > > >         >
> > > >         https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > >         
> > > >         
> > > >         _______________________________________________
> > > >         Swift-devel mailing list
> > > >         Swift-devel at ci.uchicago.edu
> > > >         https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > > 
> > > > 
> > > 
> > > 
> > > 
> 
> 
> 


From hategan at mcs.anl.gov  Sat Jun 15 21:43:54 2013
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sat, 15 Jun 2013 19:43:54 -0700
Subject: [Swift-devel] coaster transfer performance
In-Reply-To: <255121992.3510902.1371234594546.JavaMail.root@mcs.anl.gov>
References: <255121992.3510902.1371234594546.JavaMail.root@mcs.anl.gov>
Message-ID: <1371350634.20565.6.camel@echo>

I did more testing on this. I believe that the stage-out performance is
due to two factors. One is the default TCP send buffer size on beagle (a
whooping 8192 bytes) and the second is unexplained slowness of the
beagle -> swift.rcc connection.

I did a netcat (sadly the netcat on beagle doesn't allow me to specify
custom buffer sizes) and what I get is 11s for a 256MB file. I get
something somewhat similar if I increase the send buffer size in the
coaster service -> swift connection.

By contrast, the same file sent through netcat from swift.rcc to
login4.beagle takes 2 seconds.

Any ideas?

Mihael

On Fri, 2013-06-14 at 13:29 -0500, Michael Wilde wrote:
> I agree about testing; note though that you need to test on a shared
> filesystem under load before you will see where the bigger buffers
> make a difference.  For example, during times when "ls" on /home
> or /project (on midway) or /lustre (on beagle) run very slow.  Thats
> when reducing the number of trips of the file server makes the most
> difference.
> 
> - Mike
> 
> 
> ----- Original Message -----
> > From: "Mihael Hategan" <hategan at mcs.anl.gov>
> > To: "Michael Wilde" <wilde at mcs.anl.gov>
> > Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> > Sent: Friday, June 14, 2013 11:58:33 AM
> > Subject: Re: coaster transfer performance
> > 
> > I would like to point out that I have not seen a difference on
> > stage-out
> > performance when adjusting the buffer size (on the worker side). I
> > have
> > *not* tried a similar experiment for the stage-in buffer size, and I
> > believe that before we increase that permanently, we should see if it
> > does indeed make a difference.
> > 
> > Mihael
> > 
> > On Fri, 2013-06-14 at 08:19 -0500, Michael Wilde wrote:
> > > Im moving this discussion to swift-devel.
> > > 
> > > Mihael's fix to the timeout problem works very well - that problem
> > > has not recurred so far as I can tell.
> > > 
> > > We then discussed performance improvements for coaster provider
> > > staging.
> > > 
> > > Mihael suggested this:
> > > 
> > > "there should be a commented-out line in GSSChannel that says
> > > gssContext.requestConf(false); (line 85)
> > > uncomment that and re-compile, it will disable encryption"
> > > 
> > > ...and observed this:
> > > 
> > > > Disabling encryption makes the stage-ins go from about 6 MB/s to
> > > > 70-90 MB/s.
> > > > 
> > > > However, stageouts are still slow (about 4 MB/s).
> > > 
> > > I tried that, and it speeds things up a great deal. (No
> > > measurements from the fMRI demo case yet; measurements from Mihael
> > > below.)
> > > 
> > > I also tried increasing the transfer buffer size. The current
> > > buffer is 32KB.  I tried 16X (512KB) and 4X (128KB).
> > > 
> > > At 512KB, staging-in to the app goes "very fast" (again, no
> > > measures yet) but output hangs after about 25 of 100 transfers.
> > >  At 128KB, it hasn't hung yet, and goes very fast.  I too am
> > > seeing slower stage-out times from the app back to swift than
> > > stage=in times.
> > > 
> > > I'm testing on midway's shared fs right now (GPFS?) and will test
> > > on hard disk next.
> > > 
> > > But with these two fixes, things are working with very nice
> > > reliability and performance in tests so far.
> > > 
> > > The mods for these changes are below, followed by Mihael's report
> > > on testing w/o encryption.
> > > 
> > > Yadu, please integrate your coaster-provider-staging tests into the
> > > test suite and test across a range of file sizes, durations, and
> > > endpoints. Im happy to discuss this, on this list and/or in
> > > person.
> > > 
> > > Thanks,
> > > 
> > > - Mike
> > > 
> > > mid$ cat mods
> > > modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/buffers/Buffers.java
> > > modules/provider-coaster/resources/worker.pl
> > > modules/karajan/src/org/globus/cog/karajan/workflow/service/channels/GSSChannel.java
> > > mid$ svn diff
> > > Index:
> > > modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/buffers/Buffers.java
> > > ===================================================================
> > > ---
> > > modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/buffers/Buffers.java
> > >      (revision 3672)
> > > +++
> > > modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/buffers/Buffers.java
> > >      (working copy)
> > > @@ -24,7 +24,7 @@
> > >  public class Buffers extends Thread {
> > >      public static final Logger logger =
> > >      Logger.getLogger(Buffers.class);
> > >  
> > > -    public static final int ENTRY_SIZE = 32768;
> > > +  public static final int ENTRY_SIZE = 32768 * 4;
> > >      public static final int ENTRIES_PER_STREAM = 8;
> > >      public static final int MAX_ENTRIES = 1024; // 32 MB
> > >      public static final int PERFORMANCE_LOGGING_INTERVAL = 10000;
> > > Index: modules/provider-coaster/resources/worker.pl
> > > ===================================================================
> > > --- modules/provider-coaster/resources/worker.pl        (revision
> > > 3672)
> > > +++ modules/provider-coaster/resources/worker.pl        (working
> > > copy)
> > > @@ -134,7 +134,7 @@
> > >  my $JOB_COUNT = 0;
> > >  
> > >  use constant BUFSZ => 2048;
> > > -use constant IOBUFSZ => 32768;
> > > +use constant IOBUFSZ => 32768 * 4;
> > >  use constant IOBLOCKSZ => 8;
> > >  
> > >  # If true, enable a profile result that is written to the log
> > > Index:
> > > modules/karajan/src/org/globus/cog/karajan/workflow/service/channels/GSSChannel.java
> > > ===================================================================
> > > ---
> > > modules/karajan/src/org/globus/cog/karajan/workflow/service/channels/GSSChannel.java
> > >        (revision 3672)
> > > +++
> > > modules/karajan/src/org/globus/cog/karajan/workflow/service/channels/GSSChannel.java
> > >        (working copy)
> > > @@ -82,7 +82,7 @@
> > >  
> > >                                 gssContext.requestAnonymity(false);
> > >                                 gssContext.requestCredDeleg(false);
> > > -                               //gssContext.requestConf(false);
> > > +                               gssContext.requestConf(false); //
> > > Uncommented to disable encryption
> > >                                 gssContext.setOption(GSSConstants.GSS_MODE,
> > >                                 GSIConstants.MODE_SSL);
> > >                                 gssContext.setOption(GSSConstants.DELEGATION_TYPE,
> > >                                                 GSIConstants.DELEGATION_TYPE_LIMITED);
> > > mid$
> > > 
> > > 
> > > 
> > > ----- Original Message -----
> > > > From: "Mihael Hategan" <hategan at mcs.anl.gov>
> > > > To: "Michael Wilde" <wilde at mcs.anl.gov>
> > > > Sent: Thursday, June 13, 2013 10:47:18 PM
> > > > Subject: coaster transfer performance
> > > > 
> > > > Hi,
> > > > 
> > > > This is what I'm seeing. Disabling encryption makes the stage-ins
> > > > go
> > > > from about 6 MB/s to 70-90 MB/s.
> > > > 
> > > > However, stageouts are still slow (about 4 MB/s). It turns out
> > > > that
> > > > this
> > > > is not due to the shared FS. I disabled writing to disk
> > > > completely,
> > > > and
> > > > the performance is still around 6 MB/s.
> > > > 
> > > > I'll need to find out why, but probably not tonight.
> > > > 
> > > > Mihael
> > > > 
> > > > 
> > 
> > 
> > 




From wozniak at mcs.anl.gov  Sat Jun 15 23:30:16 2013
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Sat, 15 Jun 2013 23:30:16 -0500
Subject: [Swift-devel] Please help: error "
In-Reply-To: <2036566823.3625680.1371341209756.JavaMail.root@mcs.anl.gov>
References: <2036566823.3625680.1371341209756.JavaMail.root@mcs.anl.gov>
Message-ID: <51BD3F58.2080208@mcs.anl.gov>


Ok, I will try it.

On 6/15/2013 7:06 PM, Michael Wilde wrote:
> Wonderful, that fixes it - thanks very much!
>
> Justin, my swift on midway is now updated with this fix, and is the best Swift to use there for Sundays demos.
>
> $ cat ./set-swift
> PATH=/project/wilde/swift/src/0.94/cog/modules/swift/dist/swift-svn/bin:$PATH
>
> - Mike
>
>
> ----- Original Message -----
>> From: "Mihael Hategan" <hategan at mcs.anl.gov>
>> To: "Michael Wilde" <wilde at mcs.anl.gov>
>> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>, "David Kelly" <davidk at ci.uchicago.edu>
>> Sent: Saturday, June 15, 2013 6:49:32 PM
>> Subject: Re: [Swift-devel] Please help: error "
>>
>> This should now be fixed in SVN. Thanks David for the hint about the
>> coaster service change being at the root of the problem.
>>
>> Mihael
>>
>> On Sat, 2013-06-15 at 08:58 -0500, Michael Wilde wrote:
>>> I sent this help request because I thought it was causing the demo
>>> runs to fail.
>>>
>>> When David tested, he noted that the error was *not* causing any
>>> failures.
>>>
>>> Testing now, I too see that the demo now seems to be working.
>>>
>>> Its possible that I was misled by some other aspect of the run(s) I
>>> did.
>>>
>>> So we should fix this, and verify what the impact is, but for now I
>>> *think* its OK.
>>>
>>> My command line, output, dir on midway, and sites.xml is below.
>>>
>>> The log for this runs is at
>>> http://www.ci.uchicago.edu/~wilde/atlas01-20130615-1355-z6imink0.log
>>>
>>> Thanks,
>>>
>>> - Mike
>>>
>>> swift$ swift -tc.file apps.midway -sites.file t.xml atlas01.swift
>>> Swift 0.94 swift-r6549 cog-r3672 (cog modified locally)
>>>
>>> RunID: 20130615-1355-z6imink0
>>> Progress:  time: Sat, 15 Jun 2013 13:55:47 +0000
>>> Failed to configure log file name
>>> Progress:  time: Sat, 15 Jun 2013 13:55:52 +0000  Stage in:1
>>>   Submitted:3
>>> Progress:  time: Sat, 15 Jun 2013 13:55:53 +0000  Stage in:3
>>>   Active:1
>>> Final status: Sat, 15 Jun 2013 13:55:54 +0000  Finished
>>> successfully:4
>>> swift$ pwd
>>> /home/wilde/Projects/OHBM2013/svn-mcs
>>> swift$ cat t.xml
>>> <config>
>>>
>>>    <pool handle="localhost">
>>>      <execution provider="local"/>
>>>      <profile namespace="karajan" key="initialScore">10000</profile>
>>>      <filesystem provider="local"/>
>>>      <workdirectory>/scratch/midway/{env.USER}/swiftwork</workdirectory>
>>>      <profile namespace="swift" key="stagingMethod">file</profile>
>>>      <profile namespace="karajan" key="jobThrottle">.03</profile>
>>>      <!-- eg .11 -> 12 -->
>>>    </pool>
>>>
>>>    <pool handle="midway">
>>>      <execution provider="coaster" jobmanager="local:slurm"/>
>>>
>>>      <!-- Set partition and account here: -->
>>>      <profile namespace="globus" key="queue">westmere</profile>
>>>      <profile namespace="globus" key="ppn">12</profile>
>>>      <!-- <profile namespace="globus"
>>>      key="project">pi-wilde</profile> -->
>>>
>>>      <!-- Set number of jobs and nodes per job here: -->
>>>      <profile namespace="globus" key="slots">1</profile>
>>>      <profile namespace="globus" key="maxnodes">1</profile>
>>>      <profile namespace="globus" key="nodegranularity">1</profile>
>>>      <profile namespace="globus" key="jobsPerNode">12</profile> <!--
>>>      apps per node! -->
>>>      <profile namespace="karajan" key="jobThrottle">.11</profile>
>>>      <!-- eg .11 -> 12 -->
>>>
>>>      <!-- Set estimated app time (maxwalltime) and requested job
>>>      time (maxtime) here: -->
>>>      <profile namespace="globus"
>>>      key="maxWalltime">00:15:00</profile>
>>>      <profile namespace="globus" key="maxtime">1800</profile>  <!--
>>>      in seconds! -->
>>>
>>>      <!-- Set data staging model and work dir here: -->
>>>      <filesystem provider="local"/>
>>>      <workdirectory>/scratch/midway/{env.USER}/swiftwork</workdirectory>
>>>
>>>      <!-- Typically leave these constant: -->
>>>      <profile namespace="globus"
>>>      key="slurm.exclusive">false</profile>
>>>      <profile namespace="globus"
>>>      key="highOverAllocation">100</profile>
>>>      <profile namespace="globus"
>>>      key="lowOverAllocation">100</profile>
>>>      <profile namespace="karajan" key="initialScore">10000</profile>
>>>    </pool>
>>>
>>> </config>
>>> swift$
>>>
>>> ----- Original Message -----
>>>> From: "Mihael Hategan" <hategan at mcs.anl.gov>
>>>> To: "David Kelly" <davidk at ci.uchicago.edu>
>>>> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>, "Michael Wilde"
>>>> <wilde at mcs.anl.gov>
>>>> Sent: Saturday, June 15, 2013 3:47:04 AM
>>>> Subject: Re: [Swift-devel] Please help: error "
>>>>
>>>> The change you mention was meant to re-name the coaster log file
>>>> to
>>>> something containing the date rather than just coasters.log.
>>>>
>>>> It did just occur to me that this might happen when using coaster
>>>> local:x. Is that what's happening? In other words, what's in
>>>> sites.xml
>>>> for the run that prints this error?
>>>>
>>>> Mihael
>>>>
>>>> PS: the run should just continue and there should be no stack
>>>> trace
>>>> (as
>>>> David points out).
>>>>
>>>> On Fri, 2013-06-14 at 16:36 -0500, David Kelly wrote:
>>>>> I think this is related to some changes in CoasterService.java
>>>>> last
>>>>> week. A version of Swift from a few weeks ago with the same
>>>>> scripts/setup doesn't produce this message. Swift seems to
>>>>> continue
>>>>> running after this, and I don't see a stack trace.
>>>>>
>>>>> ----- Original Message -----
>>>>>
>>>>>> From: "Mihael Hategan" <hategan at mcs.anl.gov>
>>>>>> To: "Michael Wilde" <wilde at mcs.anl.gov>
>>>>>> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
>>>>>> Sent: Friday, June 14, 2013 4:28:56 PM
>>>>>> Subject: Re: [Swift-devel] Please help: error "
>>>>>> Probably a custom log4j in your classpath that does not have
>>>>>> a
>>>>>> file
>>>>>> appender.
>>>>>> Is there a stack trace?
>>>>>> Mihael
>>>>>> On Fri, 2013-06-14 at 14:57 -0500, Michael Wilde wrote:
>>>>>>> Hi All,
>>>>>>>
>>>>>>> Can anyone help me find the cause of the error below:
>>>>>>> "Failed
>>>>>>> to
>>>>>>> configure log file name"
>>>>>>>
>>>>>>> Ive never seen this before.
>>>>>>>
>>>>>>> Im getting it on midway: swift.rcc.uchicago.edu, dir
>>>>>>> /home/wilde/Projects/OHBM2013/svn:
>>>>>>>
>>>>>>> swift$ swift -config swift.properties -tc.file apps.midway
>>>>>>> genatlas.swift -d=data_2
>>>>>>> Swift 0.94 swift-r6549 cog-r3672 (cog modified locally)
>>>>>>>
>>>>>>> RunID: 20130614-1954-66k4nvfe
>>>>>>> Progress: time: Fri, 14 Jun 2013 19:55:01 +0000
>>>>>>> Failed to configure log file name
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> - Mike
>>>>>>> _______________________________________________
>>>>>>> Swift-devel mailing list
>>>>>>> Swift-devel at ci.uchicago.edu
>>>>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>>>>> _______________________________________________
>>>>>> Swift-devel mailing list
>>>>>> Swift-devel at ci.uchicago.edu
>>>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>>>> I think this is related to some changes in CoasterService.java
>>>>> last
>>>>> week. A version of Swift from a few weeks ago with the same
>>>>> scripts/setup doesn't produce this message. Swift seems to
>>>>> continue
>>>>> running after this, and I don't see a stack trace.
>>>>>
>>>>>
>>>>> ______________________________________________________________________
>>>>>          From: "Mihael Hategan" <hategan at mcs.anl.gov>
>>>>>          To: "Michael Wilde" <wilde at mcs.anl.gov>
>>>>>          Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
>>>>>          Sent: Friday, June 14, 2013 4:28:56 PM
>>>>>          Subject: Re: [Swift-devel] Please help: error "
>>>>>          
>>>>>          Probably a custom log4j in your classpath that does not
>>>>>          have a
>>>>>          file
>>>>>          appender.
>>>>>          
>>>>>          Is there a stack trace?
>>>>>          
>>>>>          Mihael
>>>>>          
>>>>>          On Fri, 2013-06-14 at 14:57 -0500, Michael Wilde wrote:
>>>>>          > Hi All,
>>>>>          >
>>>>>          > Can anyone help me find the cause of the error below:
>>>>>          "Failed to configure log file name"
>>>>>          >
>>>>>          > Ive never seen this before.
>>>>>          >
>>>>>          > Im getting it on midway: swift.rcc.uchicago.edu,
>>>>>          dir /home/wilde/Projects/OHBM2013/svn:
>>>>>          >
>>>>>          > swift$ swift -config swift.properties -tc.file
>>>>>          > apps.midway
>>>>>          genatlas.swift -d=data_2
>>>>>          > Swift 0.94 swift-r6549 cog-r3672 (cog modified
>>>>>          > locally)
>>>>>          >
>>>>>          > RunID: 20130614-1954-66k4nvfe
>>>>>          > Progress:  time: Fri, 14 Jun 2013 19:55:01 +0000
>>>>>          > Failed to configure log file name
>>>>>          >
>>>>>          > Thanks!
>>>>>          >
>>>>>          > - Mike
>>>>>          > _______________________________________________
>>>>>          > Swift-devel mailing list
>>>>>          > Swift-devel at ci.uchicago.edu
>>>>>          >
>>>>>          https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>>>>          
>>>>>          
>>>>>          _______________________________________________
>>>>>          Swift-devel mailing list
>>>>>          Swift-devel at ci.uchicago.edu
>>>>>          https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>>>>
>>>>>
>>>>
>>>>
>>
>>



From dsk at ci.uchicago.edu  Thu Jun 20 08:14:04 2013
From: dsk at ci.uchicago.edu (Daniel S. Katz)
Date: Thu, 20 Jun 2013 09:14:04 -0400
Subject: [Swift-devel] swift download or usage picture?
Message-ID: <E5D0B7FB-8673-4236-9091-D5FDF3FE7299@ci.uchicago.edu>

Does anyone have a newer version of this that I can use in a panel talk tomorrow?

Thanks,
Dan


-- 
Daniel S. Katz
University of Chicago
(773) 834-7186 (voice)
(773) 834-6818 (fax)
d.katz at ieee.org or dsk at ci.uchicago.edu
http://www.ci.uchicago.edu/~dsk/




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130620/04319558/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-1.pdf
Type: application/pdf
Size: 1675228 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130620/04319558/attachment.pdf>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130620/04319558/attachment-0001.html>

From davidk at ci.uchicago.edu  Thu Jun 20 09:02:45 2013
From: davidk at ci.uchicago.edu (David Kelly)
Date: Thu, 20 Jun 2013 09:02:45 -0500 (CDT)
Subject: [Swift-devel] swift download or usage picture?
In-Reply-To: <E5D0B7FB-8673-4236-9091-D5FDF3FE7299@ci.uchicago.edu>
Message-ID: <1600197762.6233343.1371736965974.JavaMail.root@ci.uchicago.edu>

Hi Dan, 

The most recent version of this is at http://www.ci.uchicago.edu/swift/usage/locations.php . It's based on usage rather than downloads. 

David 
----- Original Message -----

> From: "Daniel S. Katz" <dsk at ci.uchicago.edu>
> To: "swift-devel Devel" <swift-devel at ci.uchicago.edu>
> Sent: Thursday, June 20, 2013 8:14:04 AM
> Subject: [Swift-devel] swift download or usage picture?

> Does anyone have a newer version of this that I can use in a panel
> talk tomorrow?

> Thanks,
> Dan

> --

> Daniel S. Katz
> University of Chicago
> (773) 834-7186 (voice)
> (773) 834-6818 (fax)
> d.katz at ieee.org or dsk at ci.uchicago.edu
> http://www.ci.uchicago.edu/~dsk/

> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130620/145cf0c8/attachment.html>

From yadudoc1729 at gmail.com  Thu Jun 27 16:04:12 2013
From: yadudoc1729 at gmail.com (Yadu Nand)
Date: Thu, 27 Jun 2013 16:04:12 -0500
Subject: [Swift-devel] Connecting to nodes behind a login node
Message-ID: <CANa904m5KF_PPDmMxTdJYBDPg+WuqF7G+xysDePR9HZ---3_QA@mail.gmail.com>

Hi,

I am trying to use the MCS nodes (crush, stomp, thwomp...) to test a
bag-of-workstations configuration. Earlier I used a persistent
coaster-service running on one of these nodes to get tests run, but I
would much rather connect to these from swift.rcc. Is there some way
to tell swift to make the ssh-connection taking a hop over the login
node ?

I can't seem to find a method for this outlined either in the guides
or mailing-list conversations. Any help would be appreciated.

Thanks!
Yadu


From yadudoc1729 at gmail.com  Thu Jun 27 17:03:36 2013
From: yadudoc1729 at gmail.com (Yadu Nand)
Date: Thu, 27 Jun 2013 17:03:36 -0500
Subject: [Swift-devel] Connecting to nodes behind a login node
In-Reply-To: <CANa904m5KF_PPDmMxTdJYBDPg+WuqF7G+xysDePR9HZ---3_QA@mail.gmail.com>
References: <CANa904m5KF_PPDmMxTdJYBDPg+WuqF7G+xysDePR9HZ---3_QA@mail.gmail.com>
Message-ID: <CANa904=gj3-M-ptMypS0G2g-Wh9p2H3ED8CV4PKzZCf7BOMD-w@mail.gmail.com>

Thanks to David for pointing out a very simple solution for this.

I wanted to ssh to the compute nodes behind the login node directly,
and this can be done by setting up the ssh-config file to use the
ProxyCommand feature. I set up my ~/.ssh/config file on swift.rcc with
the following :
Host churn.mcs.anl.gov
    Hostname churn.mcs.anl.gov
    User yadunand
    ProxyCommand ssh -A yadunand at login.mcs.anl.gov nc %h %p 2> /dev/null
    ForwardAgent yes

Now, I can directly ssh to churn using : "ssh churn.mcs.anl.gov" from swift.rcc.

-Yadu

On Thu, Jun 27, 2013 at 4:04 PM, Yadu Nand <yadudoc1729 at gmail.com> wrote:
> Hi,
>
> I am trying to use the MCS nodes (crush, stomp, thwomp...) to test a
> bag-of-workstations configuration. Earlier I used a persistent
> coaster-service running on one of these nodes to get tests run, but I
> would much rather connect to these from swift.rcc. Is there some way
> to tell swift to make the ssh-connection taking a hop over the login
> node ?
>
> I can't seem to find a method for this outlined either in the guides
> or mailing-list conversations. Any help would be appreciated.
>
> Thanks!
> Yadu



-- 
Yadu Nand B