From wilde at mcs.anl.gov  Fri Oct  1 10:52:59 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Fri, 1 Oct 2010 09:52:59 -0600 (GMT-06:00)
Subject: [Swift-devel] Re: Need precise throttle on local provider
In-Reply-To: <1285730186.21714.4.camel@blabla2.none>
Message-ID: <921806676.572261285948379420.JavaMail.root@zimbra.anl.gov>


----- "Mihael Hategan" <hategan at mcs.anl.gov> wrote:

> Try <profile namespace="karajan" key="jobsPerCpu">1</profile>

That did not seem to work.  What does the local provider think a CPU is?

Since I have several pools (5 for my current tests) on a 4-core host (communicado), is the provider thinking that it can do 4 jobs per pool?  I need it to do one job per pool (be see my question at the end about implementing the slot concept in the provider itself).

> > But it seems like even this is not sufficient: under heavy load, Im
> > seeing a second job start on the same pool before the prior job has
> > completed (I use "mkdir" as a pseudo-mutex, and Im running on a
> local
> > filesystem under /tmp).
> 
> Explain "seeing" in the above sentence.

What I see is that more that one job runs concurrently in the same local execution provider pool.

I observe this because the job is a shell script that creates a fixed-name directory before entering its critical section and removes that directory on exiting the critical section.

If the directory exists on entry to the critical section, the job does a long sleep (42 secs) so I can see it in ps.

My ordinary locking logic (which is a successful workaround is to sleep in 1-second intervals waiting for the directory lock to free up. Under stress tests, I occasionally see a job waiting on that lock, but the retry logic works.

The locking code is:
-----
# Ready to talk to the server: send request and read response

while true; do
  mkdir $SLOTDIR/mutex
  if [ $? != 0 ]; then
    sleep 42;  # <<<<<<<<<<<< I see this sleep occasionally. Ordinarily its a sleep 1
  else
    break;
  fi
done

echo run $(pwd)/$callFile $(pwd)/$resultFile > $SLOTDIR/toR.fifo
touch $SLOTDIR/lastwrite

echo dummy stderr response 1>&2 # FIXME - testing if this is the provider staging problem (not xfering zero len stderr)

head -3 < $SLOTDIR/fromR.fifo # FIXME: Trim this down to 1 line for each call (or same # lines for each, in particular, for "quit")

rmdir $SLOTDIR/mutex
----

> But before that, try
> jobsPerCpu.
> 

Can you comment on this second question, below?

> > 
> > Second question: If you can point me to the right place, Justin or
> I
> > could do this the "right" way by modifying the local execution
> > provider set set "SLOT" numbers.  I initially thought the current
> hack
> > would be easier, and it seemed to work under standalone testing,
> but
> > seems to be failing now in the live setting.
> 
> The right way, I would think, is to modify the relevant throttling
> parameters for the scheduler for that site. That is, the local
> provider
> should not have anything to do with this.

I disagree. This "slot" functionality is the primary goal. The throttling approach is a poor workaround, as it forces me to create one local pool per core, when what I really want is just one pool.

> Luckily there already is a
> parameter to limit the number of concurrent jobs (and I mentioned it
> before).
> 
> Mihael

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From hategan at mcs.anl.gov  Fri Oct  1 11:38:50 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 01 Oct 2010 09:38:50 -0700
Subject: [Swift-devel] Re: Need precise throttle on local provider
In-Reply-To: <921806676.572261285948379420.JavaMail.root@zimbra.anl.gov>
References: <921806676.572261285948379420.JavaMail.root@zimbra.anl.gov>
Message-ID: <1285951130.5476.8.camel@blabla2.none>

On Fri, 2010-10-01 at 09:52 -0600, Michael Wilde wrote:
> ----- "Mihael Hategan" <hategan at mcs.anl.gov> wrote:
> 
> > Try <profile namespace="karajan" key="jobsPerCpu">1</profile>
> 
> That did not seem to work.  What does the local provider think a CPU is?

The provider does not meddle with scheduling and task management. But
the default is 1 cpu per host.

Though looking at the code it may be that the swift scheduler ignores
that parameter. So I will probably have to fix that.

> 
> Since I have several pools (5 for my current tests) on a 4-core host
> (communicado), is the provider thinking that it can do 4 jobs per
> pool?

Should be one job per pool.

>   I need it to do one job per pool (be see my question at the end
> about implementing the slot concept in the provider itself).
> 
> > > But it seems like even this is not sufficient: under heavy load, Im
> > > seeing a second job start on the same pool before the prior job has
> > > completed (I use "mkdir" as a pseudo-mutex, and Im running on a
> > local
> > > filesystem under /tmp).
> > 
> > Explain "seeing" in the above sentence.
> 
> What I see is that more that one job runs concurrently in the same local execution provider pool.
> 
> I observe this because the job is a shell script that creates a fixed-name directory before entering its critical section and removes that directory on exiting the critical section.
> 
> If the directory exists on entry to the critical section, the job does a long sleep (42 secs) so I can see it in ps.
> 
> My ordinary locking logic (which is a successful workaround is to sleep in 1-second intervals waiting for the directory lock to free up. Under stress tests, I occasionally see a job waiting on that lock, but the retry logic works.
> 
> The locking code is:
> -----
> # Ready to talk to the server: send request and read response
> 
> while true; do
>   mkdir $SLOTDIR/mutex
>   if [ $? != 0 ]; then
>     sleep 42;  # <<<<<<<<<<<< I see this sleep occasionally. Ordinarily its a sleep 1
>   else
>     break;
>   fi
> done
> 
> echo run $(pwd)/$callFile $(pwd)/$resultFile > $SLOTDIR/toR.fifo
> touch $SLOTDIR/lastwrite
> 
> echo dummy stderr response 1>&2 # FIXME - testing if this is the provider staging problem (not xfering zero len stderr)
> 
> head -3 < $SLOTDIR/fromR.fifo # FIXME: Trim this down to 1 line for each call (or same # lines for each, in particular, for "quit")
> 
> rmdir $SLOTDIR/mutex
> ----
> 
> > But before that, try
> > jobsPerCpu.
> > 
> 
> Can you comment on this second question, below?
> 
> > > 
> > > Second question: If you can point me to the right place, Justin or
> > I
> > > could do this the "right" way by modifying the local execution
> > > provider set set "SLOT" numbers.  I initially thought the current
> > hack
> > > would be easier, and it seemed to work under standalone testing,
> > but
> > > seems to be failing now in the live setting.
> > 
> > The right way, I would think, is to modify the relevant throttling
> > parameters for the scheduler for that site. That is, the local
> > provider
> > should not have anything to do with this.
> 
> I disagree. This "slot" functionality is the primary goal. The
> throttling approach is a poor workaround, as it forces me to create
> one local pool per core, when what I really want is just one pool.

I am unsure what one thing has to do with the other. If you say
poolCpus=4 and jobsPerCpu=1, you presumably get what you want. The issue
isn't how you express the throttling, but where in the code the
throttling is implemented. And I don't see what the mechanism used for
submission (e.g. gt2, local, pbs - i.e. provider) has to do with how to
distribute jobs to sites (the scheduler).

Mihael



From wilde at mcs.anl.gov  Sat Oct  2 07:43:40 2010
From: wilde at mcs.anl.gov (wilde at mcs.anl.gov)
Date: Sat, 2 Oct 2010 06:43:40 -0600 (GMT-06:00)
Subject: [Swift-devel] Problems in coaster block termination and restart
In-Reply-To: <272024273.613481286022888152.JavaMail.root@zimbra.anl.gov>
Message-ID: <2064660086.613511286023420170.JavaMail.root@zimbra.anl.gov>

Mihael, can you look at the Swift run on TeraPort login1 at:

/scratch/local/wilde/SwiftR/swift.local.3162

This test ran about 20 iterations of the Swift-R test battery and then hung with 5 jobs in "active" state but not completing. Then Swift finally quit when a worker cut off an executing job (as I have retries off here).

You can see this in swift stdout in swift.stdouterr in that dir.

I *think* the run hanging has something to do with coaster block termination and restart.  tc, sites.xml and swift.properties file (cf) are all in that directory.  Command line to start swift was:

$SWIFTRBIN/../swift/bin/swift -config cf -tc.file tc -sites.file sites.xml $script -pipedir=$(pwd) >& swift.stdouterr </dev/null

$script is in that dir, rserver.swift.

Ran about 2600 jobs before hanging; it went through at least 2 rounds of coaster blocks before hanging.

I got about 7 emails from PBS on walltime exceeded.  I suspect my sites.xml coaster parameters could use some tuning; its hard to determine the right time-block settings due to the dynamic and sporadic job submission rates. Specifically, in these tests, no R-evaluation job will run for more than about 15 seconds, but they get submitted in various bursts as the test proceeds; then the pattern repeats as the test is repeated.  I suspect the bursts of job concurrency range from 1 to 15 jobs; maybe a bit less at the moment.

This is pretty high prio (for the Swift R release for OpenMX). But I will try to work around it with manual coaster blocks.

Im running at or close to the latest trunk revision.

Thanks,

- Mike
ory



From wilde at mcs.anl.gov  Sat Oct  2 09:58:47 2010
From: wilde at mcs.anl.gov (wilde at mcs.anl.gov)
Date: Sat, 2 Oct 2010 08:58:47 -0600 (GMT-06:00)
Subject: [Swift-devel] Problems in coaster block termination and restart
In-Reply-To: <224841870.614731286031145733.JavaMail.root@zimbra.anl.gov>
Message-ID: <987444478.614791286031527907.JavaMail.root@zimbra.anl.gov>

Mihael, another test which encounters a similar but different problem:

This time I set the app's maxwalltime artificially higher to get longer-running coaster blocks. With this setting, 200 passes of the test battery ran perfectly, doing over 17,000 coaster jobs.

But then, after the tests completed and the swift iterate loop sat idle a while (maybe 30-60 minutes) I came back to it and started anther test loop. When the iterate woke up, swift was unable to run even the first coaster job.

You can see the logs for this in:

tp-login1.ci.uchicago.edu: /scratch/local/wilde/SwiftR/swift.local.8750

and I paste below the swift stdout that shows the problem (including a Java NPE).

Its possible that in both this run and the prior I have aggravated a coaster issue by fiddling with the worker.pl timeouts that I needed to change from prior persistent & passive coasters configs.  I'll look into that side of it.

- Mike

17,889 jobs finished OK for 200 passes of the test script.

It then sat idle a while, then I started another test:

Progress:  Finished successfully:17889
SwiftScript trace: rserver: got dir, /scratch/local/wilde/SwiftR/requests.P28845/R0002412
Progress:  Submitted:1  Finished successfully:17889

Then the timeouts and npe.

This was with coaster jobmanager local:local, so the swift log should show the coaster service activity.

- Mike

---
Progress:  Stage in:4  Submitting:1  Finished successfully:17883
Progress:  Checking status:1  Stage out:4  Finished successfully:17883
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
Progress:  Finished successfully:17889
SwiftScript trace: rserver: got dir, /scratch/local/wilde/SwiftR/requests.P28845/R0002412
Progress:  Submitted:1  Finished successfully:17889
Progress:  Submitted:1  Finished successfully:17889
Progress:  Submitted:1  Finished successfully:17889
Progress:  Submitted:1  Finished successfully:17889
Command(610, SUBMITJOB): handling reply timeout; sendReqTime=101002-093633.473, sendTime=101002-093633.474, now=101002-093833.490
Command(610, SUBMITJOB)fault was: Reply timeout
org.globus.cog.karajan.workflow.service.ReplyTimeoutException
        at org.globus.cog.karajan.workflow.service.commands.Command.handleReplyTimeout(Command.java:280)
        at org.globus.cog.karajan.workflow.service.commands.Command$Timeout.run(Command.java:285)
        at java.util.TimerThread.mainLoop(Timer.java:512)
        at java.util.TimerThread.run(Timer.java:462)
Failed to shut down channel
java.lang.NullPointerException
        at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.configureHeartBeat(AbstractKarajanChannel.java:57)
        at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.<init>(AbstractKarajanChannel.java:52)
        at org.globus.cog.karajan.workflow.service.channels.NullChannel.<init>(NullChannel.java:18)
        at org.globus.cog.karajan.workflow.service.channels.ChannelManager.unregisterChannel(ChannelManager.java:401)
        at org.globus.cog.karajan.workflow.service.channels.ChannelManager.shutdownChannel(ChannelManager.java:411)
        at org.globus.cog.karajan.workflow.service.channels.ChannelManager.handleChannelException(ChannelManager.java:284)
        at org.globus.cog.karajan.workflow.service.channels.AbstractStreamKarajanChannel.handleChannelException(AbstractStreamKarajan\
Channel.java:83)
        at org.globus.cog.karajan.workflow.service.channels.AbstractStreamKarajanChannel$Sender.run(AbstractStreamKarajanChannel.java\
:257)
Exception in thread "Sender" java.lang.NullPointerException
        at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.configureHeartBeat(AbstractKarajanChannel.java:57)
        at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.<init>(AbstractKarajanChannel.java:52)
        at org.globus.cog.karajan.workflow.service.channels.NullChannel.<init>(NullChannel.java:22)
        at org.globus.cog.karajan.workflow.service.channels.AbstractStreamKarajanChannel.handleChannelException(AbstractStreamKarajan\
Channel.java:85)
        at org.globus.cog.karajan.workflow.service.channels.AbstractStreamKarajanChannel$Sender.run(AbstractStreamKarajanChannel.java\
:257)
Progress:  Submitted:1  Finished successfully:17889
Progress:  Submitted:1  Finished successfully:17889
Progress:  Submitted:1  Finished successfully:17889
Progress:  Submitted:1  Finished successfully:17889
Progress:  Submitted:1  Finished successfully:17889

---

----- wilde at mcs.anl.gov wrote:

> Mihael, can you look at the Swift run on TeraPort login1 at:
> 
> /scratch/local/wilde/SwiftR/swift.local.3162
> 
> This test ran about 20 iterations of the Swift-R test battery and then
> hung with 5 jobs in "active" state but not completing. Then Swift
> finally quit when a worker cut off an executing job (as I have retries
> off here).
> 
> You can see this in swift stdout in swift.stdouterr in that dir.
> 
> I *think* the run hanging has something to do with coaster block
> termination and restart.  tc, sites.xml and swift.properties file (cf)
> are all in that directory.  Command line to start swift was:
> 
> $SWIFTRBIN/../swift/bin/swift -config cf -tc.file tc -sites.file
> sites.xml $script -pipedir=$(pwd) >& swift.stdouterr </dev/null
> 
> $script is in that dir, rserver.swift.
> 
> Ran about 2600 jobs before hanging; it went through at least 2 rounds
> of coaster blocks before hanging.
> 
> I got about 7 emails from PBS on walltime exceeded.  I suspect my
> sites.xml coaster parameters could use some tuning; its hard to
> determine the right time-block settings due to the dynamic and
> sporadic job submission rates. Specifically, in these tests, no
> R-evaluation job will run for more than about 15 seconds, but they get
> submitted in various bursts as the test proceeds; then the pattern
> repeats as the test is repeated.  I suspect the bursts of job
> concurrency range from 1 to 15 jobs; maybe a bit less at the moment.
> 
> This is pretty high prio (for the Swift R release for OpenMX). But I
> will try to work around it with manual coaster blocks.
> 
> Im running at or close to the latest trunk revision.
> 
> Thanks,
> 
> - Mike
> ory
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From hategan at mcs.anl.gov  Sat Oct  2 14:20:10 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sat, 02 Oct 2010 12:20:10 -0700
Subject: [Swift-devel] Re: Need to integrate SGE provider into trunk
In-Reply-To: <1285694027.18611.1.camel@blabla2.none>
References: <789287483.336151285670767227.JavaMail.root@zimbra.anl.gov>
	<1285694027.18611.1.camel@blabla2.none>
Message-ID: <1286047210.10591.0.camel@blabla2.none>

I merged the local-scheduler module from the stable branch to trunk in
cog r2893.

Mihael

On Tue, 2010-09-28 at 10:13 -0700, Mihael Hategan wrote:
> On Tue, 2010-09-28 at 04:46 -0600, Michael Wilde wrote:
> > Mihael, I just noticed that the SGE provider is in the stable branch but not in trunk.
> > 
> > Can you move it there? Is this a trivial copy or are there divergent changes in provider-localscheduler/ that need to be integrated?
> 
> I will move it there.
> 
> Btw, my schedule is shaping up. Here's the outline:
> M, W - no chance
> T, F - some time in the morning
> R, S, S - available
> 
> Mihael
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel




From hategan at mcs.anl.gov  Sat Oct  2 14:29:51 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sat, 02 Oct 2010 12:29:51 -0700
Subject: [Swift-devel] Re: Need precise throttle on local provider
In-Reply-To: <1285951130.5476.8.camel@blabla2.none>
References: <921806676.572261285948379420.JavaMail.root@zimbra.anl.gov>
	<1285951130.5476.8.camel@blabla2.none>
Message-ID: <1286047791.10677.4.camel@blabla2.none>

On Fri, 2010-10-01 at 09:38 -0700, Mihael Hategan wrote:
> > 
> > I disagree. This "slot" functionality is the primary goal. The
> > throttling approach is a poor workaround, as it forces me to create
> > one local pool per core, when what I really want is just one pool.
> 
> I am unsure what one thing has to do with the other. If you say
> poolCpus=4 and jobsPerCpu=1, you presumably get what you want. The issue
> isn't how you express the throttling, but where in the code the
> throttling is implemented. And I don't see what the mechanism used for
> submission (e.g. gt2, local, pbs - i.e. provider) has to do with how to
> distribute jobs to sites (the scheduler).

I now see better what you are saying.

I believe that there are two problems:
1. Enforcing a limited number of concurrent jobs on a site
2. Labeling the jobs in such a way that any two concurrent jobs always
have a distinct label.

I believe that this is a scenario that is very unlikely to occur in
normal swift usage. I would therefore be somewhat against writing
"generic" code to address (2) if a possibility exists to solve the
problem with the unmodified swift. Assuming (1), that possibility is
"virtual sites" which is what you are trying to do.

I don't feel strongly about this though. What do others think?

Mihael



From hategan at mcs.anl.gov  Sat Oct  2 15:48:21 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sat, 02 Oct 2010 13:48:21 -0700
Subject: [Swift-devel] Re: Need precise throttle on local provider
In-Reply-To: <179574332.422131285720167850.JavaMail.root@zimbra.anl.gov>
References: <179574332.422131285720167850.JavaMail.root@zimbra.anl.gov>
Message-ID: <1286052501.15483.5.camel@blabla2.none>

On Tue, 2010-09-28 at 18:29 -0600, wilde at mcs.anl.gov wrote:
> Mihael,
[...]

> I thought that the correct value for jobThrottle would be 0.0 to
> ensure 1 job, but from experimentation I found that I needed to set it
> to a slightly negative value, as above (-0.001).

The code was broken in two ways:
1. the check was overloaded = !(load <= maxLoad), which meant that a
host with exactly its maximum load was not considered overloaded. I
changed the comparison to "<". This means that 0.0 should now work.
2. There was some unsynchronized access to the host set (and both the
scheduler thread as well as job notification threads could concurrently
access it). While it was very unlikely for you to see the problem that
this would have caused, having only one site in sites.xml (which is what
I was testing with) did trigger it. If fixed that.

In addition I think I fixed the occasional "Failed to release host"
problem.

All this in cog r2898.

I'll test some more to see if I really get at most one job at a time.

Mihael




From wilde at mcs.anl.gov  Sun Oct  3 20:46:05 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Sun, 3 Oct 2010 19:46:05 -0600 (GMT-06:00)
Subject: [Swift-devel] Coaster provider staging data xfer problem
In-Reply-To: <1286063805.30926.2.camel@blabla2.none>
Message-ID: <1961957963.631971286156765019.JavaMail.root@zimbra.anl.gov>

I just re-ran what I thought was my failing test, and it ran OK, but failed strangely in the swift cleanup process.

Localhost coasters; provider staging.

This run is at the moment on communicado in /tmp/wilde/run01 (Im running in /tmp due to the nature of the IO)

The test is running 9999 cat jobs; one file in and one out per job.  The file sizes are on order of <500KB each (random sizes).

all 9999 files were produced, but then I got a lot of unlink messages and some strange exit code 11 error.

The messages are in swift.stdouterr

The script eas executed using ./run.sh; tc and sites file are in that run01 dir.

This is worth looking at but low prio I think. I think the script terminated cleanly on smaller runs (-n=5, -n=100). So perhaps provider staging gets confused or has sync/mutex problems related to cleanup that occur at larger volumes of file copies???

At any rate, this was *not* the error that I was referring to in the message below; in that test, staging died in the middle of a run.  I will also try to test between two hosts.

- Mike


----- "Mihael Hategan" <hategan at mcs.anl.gov> wrote:

> On Sat, 2010-10-02 at 17:51 -0600, wilde at mcs.anl.gov wrote:
> > ----- "Mihael Hategan" <hategan at mcs.anl.gov> wrote:
> > > 
> > > Ok. I'll look at that. Just to be clear, you are talking about
> > > gridftp=coaster rather than use.provider.staging, right?
> > 
> > No, I dont *think* so!
> 
> Ok.
> 
> > 
> > What I meant above was provider staging via the coaster execution
> > provider, which is the only coaster based data transport technique
> I
> > knew of.
> > 
> > I'll try to replicate my test and send it.
> 
> Ok. I tried 1024 jobs, 8 concurrent, 7MB files and I can't reproduce
> it,
> so it may not be straightforward.
> 
> > 
> > I didnt know there was such a thing as gridftp=coaster!
> > 
> > Would that be done by saying <filesystem provider="coaster"> ?
> 
> Yes.
> 
> > I didnt know you could say either of those. Can you explain what
> that
> > would do and how to say it? Is it a different data provider path
> than
> > provider staging, but which still uses coasters?  Independent of
> > coaster execution? (I might be way off base here, sorry!)
> 
> Yes, and yes. Though I recommend provider staging.
> 
> Mihael

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From wilde at mcs.anl.gov  Sun Oct  3 21:20:05 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Sun, 3 Oct 2010 20:20:05 -0600 (GMT-06:00)
Subject: [Swift-devel] Coaster provider staging data xfer problem
In-Reply-To: <1961957963.631971286156765019.JavaMail.root@zimbra.anl.gov>
Message-ID: <1102522233.632501286158805329.JavaMail.root@zimbra.anl.gov>

Interesting: at 5000 jobs, the run completes normally; at 10,000 it fails again, as before. (I adjusted the script slightly to give 10,000 jobs instead of 9999)

- Mike


----- "Michael Wilde" <wilde at mcs.anl.gov> wrote:

> I just re-ran what I thought was my failing test, and it ran OK, but
> failed strangely in the swift cleanup process.
> 
> Localhost coasters; provider staging.
> 
> This run is at the moment on communicado in /tmp/wilde/run01 (Im
> running in /tmp due to the nature of the IO)
> 
> The test is running 9999 cat jobs; one file in and one out per job. 
> The file sizes are on order of <500KB each (random sizes).
> 
> all 9999 files were produced, but then I got a lot of unlink messages
> and some strange exit code 11 error.
> 
> The messages are in swift.stdouterr
> 
> The script eas executed using ./run.sh; tc and sites file are in that
> run01 dir.
> 
> This is worth looking at but low prio I think. I think the script
> terminated cleanly on smaller runs (-n=5, -n=100). So perhaps provider
> staging gets confused or has sync/mutex problems related to cleanup
> that occur at larger volumes of file copies???
> 
> At any rate, this was *not* the error that I was referring to in the
> message below; in that test, staging died in the middle of a run.  I
> will also try to test between two hosts.
> 
> - Mike
> 
> 
> ----- "Mihael Hategan" <hategan at mcs.anl.gov> wrote:
> 
> > On Sat, 2010-10-02 at 17:51 -0600, wilde at mcs.anl.gov wrote:
> > > ----- "Mihael Hategan" <hategan at mcs.anl.gov> wrote:
> > > > 
> > > > Ok. I'll look at that. Just to be clear, you are talking about
> > > > gridftp=coaster rather than use.provider.staging, right?
> > > 
> > > No, I dont *think* so!
> > 
> > Ok.
> > 
> > > 
> > > What I meant above was provider staging via the coaster execution
> > > provider, which is the only coaster based data transport
> technique
> > I
> > > knew of.
> > > 
> > > I'll try to replicate my test and send it.
> > 
> > Ok. I tried 1024 jobs, 8 concurrent, 7MB files and I can't
> reproduce
> > it,
> > so it may not be straightforward.
> > 
> > > 
> > > I didnt know there was such a thing as gridftp=coaster!
> > > 
> > > Would that be done by saying <filesystem provider="coaster"> ?
> > 
> > Yes.
> > 
> > > I didnt know you could say either of those. Can you explain what
> > that
> > > would do and how to say it? Is it a different data provider path
> > than
> > > provider staging, but which still uses coasters?  Independent of
> > > coaster execution? (I might be way off base here, sorry!)
> > 
> > Yes, and yes. Though I recommend provider staging.
> > 
> > Mihael
> 
> -- 
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From hategan at mcs.anl.gov  Sun Oct  3 21:32:17 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 03 Oct 2010 19:32:17 -0700
Subject: [Swift-devel] Coaster provider staging data xfer problem
In-Reply-To: <1102522233.632501286158805329.JavaMail.root@zimbra.anl.gov>
References: <1102522233.632501286158805329.JavaMail.root@zimbra.anl.gov>
Message-ID: <1286159537.4925.0.camel@blabla2.none>


On Sun, 2010-10-03 at 20:20 -0600, Michael Wilde wrote:
> Interesting: at 5000 jobs, the run completes normally; at 10,000 it
> fails again, as before. (I adjusted the script slightly to give 10,000
> jobs instead of 9999)

Strange indeed. Does it depend on file size?




From wilde at mcs.anl.gov  Sun Oct  3 23:05:52 2010
From: wilde at mcs.anl.gov (wilde at mcs.anl.gov)
Date: Sun, 3 Oct 2010 22:05:52 -0600 (GMT-06:00)
Subject: [Swift-devel] Coaster provider staging data xfer problem
In-Reply-To: <1327139875.634231286165123525.JavaMail.root@zimbra.anl.gov>
Message-ID: <1524506201.634251286165152328.JavaMail.root@zimbra.anl.gov>

Hard to say; probably not on individual filesize; perhaps on cumulative size?

The file sizes are in a fixed directory, generated once (for all the tests Ive done)
sizes range from about 1K to 300K, randomly generated using $RANDOM:

10 smallest:

911
976
1752
1856
2236
2380
2612
2695
3087

10 largest:

355783
356035
356368
356402
356543
356601
357104
357117
358513

The files were generated with:

dd if=/home/wilde/swift/lab/48M of=data bs=1M
for i in $(seq -f %04g 0 9999) ; do
  dd if=data of=f.$i.in bs=$(($RANDOM*10+$RANDOM)) count=1
done


- Mike

----- "Mihael Hategan" <hategan at mcs.anl.gov> wrote:

> On Sun, 2010-10-03 at 20:20 -0600, Michael Wilde wrote:
> > Interesting: at 5000 jobs, the run completes normally; at 10,000 it
> > fails again, as before. (I adjusted the script slightly to give
> 10,000
> > jobs instead of 9999)
> 
> Strange indeed. Does it depend on file size?

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From wozniak at mcs.anl.gov  Mon Oct  4 10:00:50 2010
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Mon, 4 Oct 2010 10:00:50 -0500 (CDT)
Subject: [Swift-devel] Re: Need precise throttle on local provider
In-Reply-To: <1286047791.10677.4.camel@blabla2.none>
References: <921806676.572261285948379420.JavaMail.root@zimbra.anl.gov>
	<1285951130.5476.8.camel@blabla2.none>
	<1286047791.10677.4.camel@blabla2.none>
Message-ID: <alpine.DEB.2.00.1010040948040.1756@wozniak-desktop.mcs.anl.gov>

On Sat, 2 Oct 2010, Mihael Hategan wrote:

> On Fri, 2010-10-01 at 09:38 -0700, Mihael Hategan wrote:
>>>
>>> I disagree. This "slot" functionality is the primary goal. The
>>> throttling approach is a poor workaround, as it forces me to create
>>> one local pool per core, when what I really want is just one pool.

> 1. Enforcing a limited number of concurrent jobs on a site
> 2. Labeling the jobs in such a way that any two concurrent jobs always
> have a distinct label.
>
> I believe that this is a scenario that is very unlikely to occur in
> normal swift usage. I would therefore be somewhat against writing
> "generic" code to address (2) if a possibility exists to solve the
> problem with the unmodified swift. Assuming (1), that possibility is
> "virtual sites" which is what you are trying to do.
>
> I don't feel strongly about this though. What do others think?

I think it's definitely useful in the application Mike is working on and 
could be useful in a lot of other settings.  In a small way, it exposes 
the fact that there is a non-trivial topology to work with in multicore 
settings, and resources on that SMP may need to be coordinated.  For 
example, it could be used to manage access to the LFS or distributed 
database implementation.

-- 
Justin M Wozniak


From chaturanganamal at gmail.com  Tue Oct  5 11:19:44 2010
From: chaturanganamal at gmail.com (Chaturanga Wimalarathne)
Date: Tue, 5 Oct 2010 21:49:44 +0530
Subject: [Swift-devel] Hi all,
Message-ID: <AANLkTim5iTrNnLE723NCBBWZH=kvJGXcCGuwKyw0pG6F@mail.gmail.com>

Can you please tell me is there any opportunity for me to work regarding
Swift Script language....
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101005/5f47c0a9/attachment.html>

From skenny at uchicago.edu  Tue Oct  5 12:26:18 2010
From: skenny at uchicago.edu (Sarah Kenny)
Date: Tue, 5 Oct 2010 12:26:18 -0500
Subject: [Swift-devel] new commits
Message-ID: <AANLkTimPsYeivXNZzj92ArWcPi1onU2C9D6ReVT28Z_J@mail.gmail.com>

hi all, i have a fix to swift that allows the jobName parameter to be passed
to GRAM (so the job name actually shows up in the queue). this will be my
first commit to the cog/swift source so i want to know where the best place
is to do this? i'd like it to be in the stable release so that it's there
when we update. my user group (HNL) has been using the stable branch with
this fix added for several months so it's pretty well-tested.

i guess the question is would it be best to commit this to the development
code (which as i understand is cogkit/trunk/current) to be migrated to the
stable code (cogkit/branches/4.1.7) in the near future when we decide to
update the stable branch?

also, i'm guessing i'll need to get access (from mihael or mike wilde?) to
the cog source...

~sk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101005/a24c0f0c/attachment.html>

From bugzilla-daemon at mcs.anl.gov  Tue Oct  5 12:47:25 2010
From: bugzilla-daemon at mcs.anl.gov (bugzilla-daemon at mcs.anl.gov)
Date: Tue,  5 Oct 2010 12:47:25 -0500 (CDT)
Subject: [Swift-devel] [Bug 175] ambiguous error when input file not found
In-Reply-To: <bug-175-21@http.bugzilla.mcs.anl.gov/swift/>
References: <bug-175-21@http.bugzilla.mcs.anl.gov/swift/>
Message-ID: <20101005174725.439002CD3F@wind.mcs.anl.gov>

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=175


skenny <skenny at uchicago.edu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED




-- 
Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.


From wilde at mcs.anl.gov  Tue Oct  5 12:59:01 2010
From: wilde at mcs.anl.gov (wilde at mcs.anl.gov)
Date: Tue, 5 Oct 2010 11:59:01 -0600 (GMT-06:00)
Subject: [Swift-devel] Re: Need precise throttle on local provider
In-Reply-To: <588360838.714651286299728515.JavaMail.root@zimbra.anl.gov>
Message-ID: <929801065.717261286301541168.JavaMail.root@zimbra.anl.gov>


----- "Mihael Hategan" <hategan at mcs.anl.gov> wrote:


> I now see better what you are saying.
> 
> I believe that there are two problems:
> 1. Enforcing a limited number of concurrent jobs on a site
> 2. Labeling the jobs in such a way that any two concurrent jobs
> always
> have a distinct label.
> 
> I believe that this is a scenario that is very unlikely to occur in
> normal swift usage. I would therefore be somewhat against writing
> "generic" code to address (2) if a possibility exists to solve the
> problem with the unmodified swift. Assuming (1), that possibility is
> "virtual sites" which is what you are trying to do.
> 
> I don't feel strongly about this though. What do others think?
> 
> Mihael

As Ive had the coaster slot-labelling code running in worker.pl for quite a while, Im committing it into trunk now. Its been very costly for me to keep re-integrating it as worker.pl evolves.  If better ideas on how to do this come up, we can revise or remove it.

This code sets SWIFT_JOB_SLOT and SWOFT_WORKER_PID in the environment of executed app() jobs.

I'll also commit a change to coasters build.xml to copy worker.pl to bin/ and mark it executable.

- Mike



-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From hategan at mcs.anl.gov  Tue Oct  5 12:58:33 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 05 Oct 2010 10:58:33 -0700
Subject: [Swift-devel] new commits
In-Reply-To: <AANLkTimPsYeivXNZzj92ArWcPi1onU2C9D6ReVT28Z_J@mail.gmail.com>
References: <AANLkTimPsYeivXNZzj92ArWcPi1onU2C9D6ReVT28Z_J@mail.gmail.com>
Message-ID: <1286301513.15904.2.camel@blabla2.none>

I think I committed some changes based on your patch. One problem was
that gram2 doesn't seem to support a job name. Did you find a way around
that?


On Tue, 2010-10-05 at 12:26 -0500, Sarah Kenny wrote:
> hi all, i have a fix to swift that allows the jobName parameter to be
> passed to GRAM (so the job name actually shows up in the queue). this
> will be my first commit to the cog/swift source so i want to know
> where the best place is to do this? i'd like it to be in the stable
> release so that it's there when we update. my user group (HNL) has
> been using the stable branch with this fix added for several months so
> it's pretty well-tested. 
> 
> i guess the question is would it be best to commit this to the
> development code (which as i understand is cogkit/trunk/current) to be
> migrated to the stable code (cogkit/branches/4.1.7) in the near future
> when we decide to update the stable branch?
> 
> also, i'm guessing i'll need to get access (from mihael or mike
> wilde?) to the cog source...

I can do that. Do you have a sourceforge account?




From hategan at mcs.anl.gov  Tue Oct  5 13:00:28 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 05 Oct 2010 11:00:28 -0700
Subject: [Swift-devel] Re: Need precise throttle on local provider
In-Reply-To: <929801065.717261286301541168.JavaMail.root@zimbra.anl.gov>
References: <929801065.717261286301541168.JavaMail.root@zimbra.anl.gov>
Message-ID: <1286301628.15904.4.camel@blabla2.none>

On Tue, 2010-10-05 at 11:59 -0600, wilde at mcs.anl.gov wrote:

> As Ive had the coaster slot-labelling code running in worker.pl for
> quite a while, Im committing it into trunk now. Its been very costly
> for me to keep re-integrating it as worker.pl evolves.  If better
> ideas on how to do this come up, we can revise or remove it.
> 
> This code sets SWIFT_JOB_SLOT and SWOFT_WORKER_PID in the environment
> of executed app() jobs.

Right. I see no reason why those couldn't be there. But if you commit it
to coasters, can you change to COASTER_JOB_SLOT?

> 
> I'll also commit a change to coasters build.xml to copy worker.pl to
> bin/ and mark it executable.

Why is that needed?




From skenny at uchicago.edu  Tue Oct  5 13:05:38 2010
From: skenny at uchicago.edu (Sarah Kenny)
Date: Tue, 5 Oct 2010 13:05:38 -0500
Subject: [Swift-devel] [Bug 175] ambiguous error when input file not found
In-Reply-To: <20101005174725.439002CD3F@wind.mcs.anl.gov>
References: <bug-175-21@http.bugzilla.mcs.anl.gov/swift/>
	<20101005174725.439002CD3F@wind.mcs.anl.gov>
Message-ID: <AANLkTi=YcEuXD6f+5=5V61urc01LgZ5WNmrwtCPASK=f@mail.gmail.com>

 fyi, i'm doing a little clean-up in bugzilla...

On Tue, Oct 5, 2010 at 12:47 PM, <bugzilla-daemon at mcs.anl.gov> wrote:

> https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=175
>
>
> skenny <skenny at uchicago.edu> changed:
>
>           What    |Removed                     |Added
>
> ----------------------------------------------------------------------------
>             Status|NEW                         |RESOLVED
>         Resolution|                            |FIXED
>
>
>
>
> --
> Configure bugmail:
> https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email
> ------- You are receiving this mail because: -------
> You are watching the assignee of the bug.
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101005/509f2477/attachment.html>

From wilde at mcs.anl.gov  Tue Oct  5 13:12:00 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Tue, 5 Oct 2010 12:12:00 -0600 (GMT-06:00)
Subject: [Swift-devel] Hi all,
In-Reply-To: <AANLkTim5iTrNnLE723NCBBWZH=kvJGXcCGuwKyw0pG6F@mail.gmail.com>
Message-ID: <2062629697.718171286302320231.JavaMail.root@zimbra.anl.gov>

Hi Chaturanga,

Sorry for the delayed response.  Yes, you and your fellow students should feel free to  work on some of the projects on the GSoC list, or similar ones.

The project related to making Swift use a catalog of distributed file replicas as the basis for its data management is an interesting one. I can share with you some discussion I had with other students on this. To do it in a user-ready manner is a hard project, but some interesting parts of it can be developed as a class project.

Another project is to add associative arrays to Swift, so that in addition to saying i = values[j] you can also say i = values["someKey"] - ie, permit string indices.

A third project would be to re-work the mapper naming conventions to make code more readable and less verbose, and to fix some broken mapper semantics.

A fourth project might be to make some simple enhancements to the import() facility.

Lastly, adding namespaces to Swift would be very interesting, but perhaps very difficult as well.

I and others on the list can try to help, but please be aware that our time is often very limited and our response time may be slow or non-existent at times.

Justin, Mihael, and others will likely have additional ideas, and perhaps better judgement on the feasibility of various projects.

Regards,

Mike

----- "Chaturanga Wimalarathne" <chaturanganamal at gmail.com> wrote:

> Can you please tell me is there any opportunity for me to work
> regarding Swift Script language.... 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From wilde at mcs.anl.gov  Tue Oct  5 13:16:00 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Tue, 5 Oct 2010 12:16:00 -0600 (GMT-06:00)
Subject: [Swift-devel] Re: Need precise throttle on local provider
In-Reply-To: <1286301628.15904.4.camel@blabla2.none>
Message-ID: <653935577.718381286302560456.JavaMail.root@zimbra.anl.gov>


----- "Mihael Hategan" <hategan at mcs.anl.gov> wrote:

> On Tue, 2010-10-05 at 11:59 -0600, wilde at mcs.anl.gov wrote:
> 
> > As Ive had the coaster slot-labelling code running in worker.pl for
> > quite a while, Im committing it into trunk now. Its been very
> costly
> > for me to keep re-integrating it as worker.pl evolves.  If better
> > ideas on how to do this come up, we can revise or remove it.
> > 
> > This code sets SWIFT_JOB_SLOT and SWOFT_WORKER_PID in the
> environment
> > of executed app() jobs.
> 
> Right. I see no reason why those couldn't be there. But if you commit
> it
> to coasters, can you change to COASTER_JOB_SLOT?

In my application thats not feasible, as the application scripts should work the same (and do) regardless of whether they were run under coasters (where the provider sets the slot) or under the local exec provider (where the slot is set by using multiple sites and the ENV profile).

> 
> > 
> > I'll also commit a change to coasters build.xml to copy worker.pl
> to
> > bin/ and mark it executable.
> 
> Why is that needed?

Many of us (well, Allan and I) are executing worker.pl for persistent and passive mode configurations. Its much better to get worker.pl from bin/ for these applications rather than manually fishing it out of the resources/ directory.

- Mike

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From wilde at mcs.anl.gov  Tue Oct  5 13:28:52 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Tue, 5 Oct 2010 12:28:52 -0600 (GMT-06:00)
Subject: [Swift-devel] Requests for improvements to error messages and
	logging
Message-ID: <1074761391.719211286303332355.JavaMail.root@zimbra.anl.gov>

Sarah will be shifting some of her effort (~40%) from CNARI to focus on Swift development.

Her focus is going to be on error message and logging improvement: anything related to helping the user quickly and easily understand what went wrong in their script and correct the problem.  Broadly, I think status monitoring (eg the progress output and -tui displays) fall into this category as well, as does log formatting and certainly error message content.

I'd like to suggest we use bugzilla for tracking requests for such improvements (and in general get bugzilla more firmly into our process).

In the meantime, some candidates I'd like to propose for improvement:

- put simple friendly timestamps on the progress indicators

- snap more info from cmd line, sites, tc, and swift.properties into the log

- make stdout/err and job failure info go to the log as soon as errors occur, rather than buffering them to show up at the end of the log (as users often ^C out of swift and never see these errors)

- diagnose the error codes 1 and 254 error messages that occur on the execution site at run time, and produce better error messages.

- create a log filter to lift the salient info about job execution out of the log into a simple readable stream that would make sense to users at all levels of expertise.  Ie, what was transfered to and executed where, when, and for how long.

This is going to be a long process and will need a lot of input, but I wanted to provide you some places to start focusing on, Sarah.

- Mike


From skenny at uchicago.edu  Tue Oct  5 14:31:00 2010
From: skenny at uchicago.edu (Sarah Kenny)
Date: Tue, 5 Oct 2010 14:31:00 -0500
Subject: [Swift-devel] new commits
In-Reply-To: <1286301513.15904.2.camel@blabla2.none>
References: <AANLkTimPsYeivXNZzj92ArWcPi1onU2C9D6ReVT28Z_J@mail.gmail.com>
	<1286301513.15904.2.camel@blabla2.none>
Message-ID: <AANLkTimcKpH-4fKA8DM_kARAGybph+wsEbkyRtuvwe7m@mail.gmail.com>

On Tue, Oct 5, 2010 at 12:58 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> I think I committed some changes based on your patch. One problem was
> that gram2 doesn't seem to support a job name. Did you find a way around
> that?
>

hmm, it is working on abe (gram2) and queenbee (gram5)...were you getting an
error running it elsewhere or did it just not pick up the name? i'm thinking
if it works for some gram versions and not others it might be ok to commit
unless it actually causes an error in versions where the name is not passed
(?)

i thought you had committed (or were going to commit) the change as well,
but i'm not seeing it here:

http://cogkit.svn.sourceforge.net/viewvc/cogkit/branches/4.1.7/src/cog/modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/Settings.java?revision=2623&view=markup&pathrev=2623

is there somewhere else i should be looking?


>
>
> On Tue, 2010-10-05 at 12:26 -0500, Sarah Kenny wrote:
> > hi all, i have a fix to swift that allows the jobName parameter to be
> > passed to GRAM (so the job name actually shows up in the queue). this
> > will be my first commit to the cog/swift source so i want to know
> > where the best place is to do this? i'd like it to be in the stable
> > release so that it's there when we update. my user group (HNL) has
> > been using the stable branch with this fix added for several months so
> > it's pretty well-tested.
> >
> > i guess the question is would it be best to commit this to the
> > development code (which as i understand is cogkit/trunk/current) to be
> > migrated to the stable code (cogkit/branches/4.1.7) in the near future
> > when we decide to update the stable branch?
> >
> > also, i'm guessing i'll need to get access (from mihael or mike
> > wilde?) to the cog source...
>
> I can do that. Do you have a sourceforge account?
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101005/3b654731/attachment.html>

From skenny at uchicago.edu  Tue Oct  5 14:33:15 2010
From: skenny at uchicago.edu (Sarah Kenny)
Date: Tue, 5 Oct 2010 14:33:15 -0500
Subject: [Swift-devel] editing swift user guide
Message-ID: <AANLkTimH6rCmxfLisqXc+Vm_3MNrDzGQBU3+kmKrHJTb@mail.gmail.com>

how can i edit the swift user guide? i'm seeing a few old bugs in bugzilla
that could be cleared up with a very minor change to the doc...

~sk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101005/47ec86d3/attachment.html>

From wilde at mcs.anl.gov  Tue Oct  5 14:50:04 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Tue, 5 Oct 2010 13:50:04 -0600 (GMT-06:00)
Subject: [Swift-devel] editing swift user guide
In-Reply-To: <AANLkTimH6rCmxfLisqXc+Vm_3MNrDzGQBU3+kmKrHJTb@mail.gmail.com>
Message-ID: <1624213036.725411286308204739.JavaMail.root@zimbra.anl.gov>

Justin has been looking for a way to:

- have an online doc version per-release and per-branch

- easily svn checkout the docs, test your changes to make sure they look ready to post, commit, and push the changes to the right online version to match that branch and release you are editing

Sarah, can you coordinate with Justin on this, post the proposed method to the list, and make it happen?

Thanks,

- Mike


----- "Sarah Kenny" <skenny at uchicago.edu> wrote:

> how can i edit the swift user guide? i'm seeing a few old bugs in
> bugzilla that could be cleared up with a very minor change to the
> doc...
> 
> ~sk
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From skenny at uchicago.edu  Tue Oct  5 15:08:39 2010
From: skenny at uchicago.edu (Sarah Kenny)
Date: Tue, 5 Oct 2010 15:08:39 -0500
Subject: [Swift-devel] editing swift user guide
In-Reply-To: <1624213036.725411286308204739.JavaMail.root@zimbra.anl.gov>
References: <AANLkTimH6rCmxfLisqXc+Vm_3MNrDzGQBU3+kmKrHJTb@mail.gmail.com>
	<1624213036.725411286308204739.JavaMail.root@zimbra.anl.gov>
Message-ID: <AANLkTikD61y+YN+QvQGqx+aZ=dNgAwXVoVff3zcLx3AK@mail.gmail.com>

does anyone know how/where it's currently maintained?

~sk

On Tue, Oct 5, 2010 at 2:50 PM, Michael Wilde <wilde at mcs.anl.gov> wrote:

> Justin has been looking for a way to:
>
> - have an online doc version per-release and per-branch
>
> - easily svn checkout the docs, test your changes to make sure they look
> ready to post, commit, and push the changes to the right online version to
> match that branch and release you are editing
>
> Sarah, can you coordinate with Justin on this, post the proposed method to
> the list, and make it happen?
>
> Thanks,
>
> - Mike
>
>
> ----- "Sarah Kenny" <skenny at uchicago.edu> wrote:
>
> > how can i edit the swift user guide? i'm seeing a few old bugs in
> > bugzilla that could be cleared up with a very minor change to the
> > doc...
> >
> > ~sk
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101005/ca1ede5b/attachment.html>

From bugzilla-daemon at mcs.anl.gov  Tue Oct  5 15:30:05 2010
From: bugzilla-daemon at mcs.anl.gov (bugzilla-daemon at mcs.anl.gov)
Date: Tue,  5 Oct 2010 15:30:05 -0500 (CDT)
Subject: [Swift-devel] [Bug 22] configurable remote filesystem layout
In-Reply-To: <bug-22-21@http.bugzilla.mcs.anl.gov/swift/>
References: <bug-22-21@http.bugzilla.mcs.anl.gov/swift/>
Message-ID: <20101005203005.65E2C2CD55@wind.mcs.anl.gov>

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=22


skenny <skenny at uchicago.edu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |skenny at uchicago.edu
         Resolution|                            |FIXED




--- Comment #5 from skenny <skenny at uchicago.edu>  2010-10-05 15:30:05 ---
option (i) listed here has been in place for some time, so i am closing out
this ticket.

-- 
Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching the reporter.


From wilde at mcs.anl.gov  Tue Oct  5 16:02:19 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Tue, 5 Oct 2010 15:02:19 -0600 (GMT-06:00)
Subject: [Swift-devel] editing swift user guide
In-Reply-To: <AANLkTikD61y+YN+QvQGqx+aZ=dNgAwXVoVff3zcLx3AK@mail.gmail.com>
Message-ID: <443386466.731101286312539713.JavaMail.root@zimbra.anl.gov>

There are some notes on maintaining the Web and User Guide here:
  http://www.ci.uchicago.edu/wiki/bin/view/SWFT/MaintainingSwiftWebConten

Please update that page as you enhance things.

Justin and/or David might be able to provide more details.

Thanks,

Mike

----- "Sarah Kenny" <skenny at uchicago.edu> wrote:

> does anyone know how/where it's currently maintained?
> 
> ~sk
> 
> 
> On Tue, Oct 5, 2010 at 2:50 PM, Michael Wilde < wilde at mcs.anl.gov >
> wrote:
> 
> 
> Justin has been looking for a way to:
> 
> - have an online doc version per-release and per-branch
> 
> - easily svn checkout the docs, test your changes to make sure they
> look ready to post, commit, and push the changes to the right online
> version to match that branch and release you are editing
> 
> Sarah, can you coordinate with Justin on this, post the proposed
> method to the list, and make it happen?
> 
> Thanks,
> 
> - Mike
> 
> 
> 
> 
> 
> ----- "Sarah Kenny" < skenny at uchicago.edu > wrote:
> 
> > how can i edit the swift user guide? i'm seeing a few old bugs in
> > bugzilla that could be cleared up with a very minor change to the
> > doc...
> >
> > ~sk
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> 
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From wozniak at mcs.anl.gov  Tue Oct  5 16:06:23 2010
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Tue, 5 Oct 2010 16:06:23 -0500 (CDT)
Subject: [Swift-devel] Re: Need precise throttle on local provider
In-Reply-To: <653935577.718381286302560456.JavaMail.root@zimbra.anl.gov>
References: <653935577.718381286302560456.JavaMail.root@zimbra.anl.gov>
Message-ID: <alpine.DEB.2.00.1010051603470.2548@wozniak-desktop.mcs.anl.gov>

On Tue, 5 Oct 2010, Michael Wilde wrote:

> ----- "Mihael Hategan" <hategan at mcs.anl.gov> wrote:
>
>> On Tue, 2010-10-05 at 11:59 -0600, wilde at mcs.anl.gov wrote:
>>
>>> This code sets SWIFT_JOB_SLOT and SWOFT_WORKER_PID in the environment 
>>> of executed app() jobs.
>>
>> Right. I see no reason why those couldn't be there. But if you commit 
>> it to coasters, can you change to COASTER_JOB_SLOT?
>
> In my application thats not feasible, as the application scripts should 
> work the same (and do) regardless of whether they were run under 
> coasters (where the provider sets the slot) or under the local exec 
> provider (where the slot is set by using multiple sites and the ENV 
> profile).

COG_SMP_SLOT ?

-- 
Justin M Wozniak


From hategan at mcs.anl.gov  Tue Oct  5 16:13:03 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 05 Oct 2010 14:13:03 -0700
Subject: [Swift-devel] new commits
In-Reply-To: <AANLkTimcKpH-4fKA8DM_kARAGybph+wsEbkyRtuvwe7m@mail.gmail.com>
References: <AANLkTimPsYeivXNZzj92ArWcPi1onU2C9D6ReVT28Z_J@mail.gmail.com>
	<1286301513.15904.2.camel@blabla2.none>
	<AANLkTimcKpH-4fKA8DM_kARAGybph+wsEbkyRtuvwe7m@mail.gmail.com>
Message-ID: <1286313183.16770.1.camel@blabla2.none>

On Tue, 2010-10-05 at 14:31 -0500, Sarah Kenny wrote:
> On Tue, Oct 5, 2010 at 12:58 PM, Mihael Hategan <hategan at mcs.anl.gov>
> wrote:
>         I think I committed some changes based on your patch. One
>         problem was
>         that gram2 doesn't seem to support a job name. Did you find a
>         way around
>         that?
> 
> hmm, it is working on abe (gram2) and queenbee (gram5)...were you
> getting an error running it elsewhere or did it just not pick up the
> name? i'm thinking if it works for some gram versions and not others
> it might be ok to commit unless it actually causes an error in
> versions where the name is not passed (?)

That was the problem. It caused an error.
> 
> i thought you had committed (or were going to commit) the change as
> well, but i'm not seeing it here:
> 
> http://cogkit.svn.sourceforge.net/viewvc/cogkit/branches/4.1.7/src/cog/modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/Settings.java?revision=2623&view=markup&pathrev=2623

It's in trunk.

But:
http://cogkit.svn.sourceforge.net/viewvc/cogkit/trunk/current/src/cog/modules/provider-gt2/src/org/globus/cog/abstraction/impl/execution/gt2/JobSubmissionTaskHandler.java?r1=2815&r2=2867

> 
> is there somewhere else i should be looking?
>  
>         
>         
>         
>         On Tue, 2010-10-05 at 12:26 -0500, Sarah Kenny wrote:
>         > hi all, i have a fix to swift that allows the jobName
>         parameter to be
>         > passed to GRAM (so the job name actually shows up in the
>         queue). this
>         > will be my first commit to the cog/swift source so i want to
>         know
>         > where the best place is to do this? i'd like it to be in the
>         stable
>         > release so that it's there when we update. my user group
>         (HNL) has
>         > been using the stable branch with this fix added for several
>         months so
>         > it's pretty well-tested.
>         >
>         > i guess the question is would it be best to commit this to
>         the
>         > development code (which as i understand is
>         cogkit/trunk/current) to be
>         > migrated to the stable code (cogkit/branches/4.1.7) in the
>         near future
>         > when we decide to update the stable branch?
>         >
>         > also, i'm guessing i'll need to get access (from mihael or
>         mike
>         > wilde?) to the cog source...
>         
>         
>         I can do that. Do you have a sourceforge account?
>         
>         
> 




From hategan at mcs.anl.gov  Tue Oct  5 16:14:34 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 05 Oct 2010 14:14:34 -0700
Subject: [Swift-devel] Re: Need precise throttle on local provider
In-Reply-To: <alpine.DEB.2.00.1010051603470.2548@wozniak-desktop.mcs.anl.gov>
References: <653935577.718381286302560456.JavaMail.root@zimbra.anl.gov>
	<alpine.DEB.2.00.1010051603470.2548@wozniak-desktop.mcs.anl.gov>
Message-ID: <1286313274.16770.2.camel@blabla2.none>

On Tue, 2010-10-05 at 16:06 -0500, Justin M Wozniak wrote:
> On Tue, 5 Oct 2010, Michael Wilde wrote:
> 
> > ----- "Mihael Hategan" <hategan at mcs.anl.gov> wrote:
> >
> >> On Tue, 2010-10-05 at 11:59 -0600, wilde at mcs.anl.gov wrote:
> >>
> >>> This code sets SWIFT_JOB_SLOT and SWOFT_WORKER_PID in the environment 
> >>> of executed app() jobs.
> >>
> >> Right. I see no reason why those couldn't be there. But if you commit 
> >> it to coasters, can you change to COASTER_JOB_SLOT?
> >
> > In my application thats not feasible, as the application scripts should 
> > work the same (and do) regardless of whether they were run under 
> > coasters (where the provider sets the slot) or under the local exec 
> > provider (where the slot is set by using multiple sites and the ENV 
> > profile).
> 
> COG_SMP_SLOT ?
> 

Sounds more appropriate, but whatever works for you.

Mihael



From dk0966 at cs.ship.edu  Tue Oct  5 16:32:15 2010
From: dk0966 at cs.ship.edu (David Kelly)
Date: Tue, 5 Oct 2010 17:32:15 -0400
Subject: [Swift-devel] editing swift user guide
In-Reply-To: <AANLkTikD61y+YN+QvQGqx+aZ=dNgAwXVoVff3zcLx3AK@mail.gmail.com>
References: <AANLkTimH6rCmxfLisqXc+Vm_3MNrDzGQBU3+kmKrHJTb@mail.gmail.com>
	<1624213036.725411286308204739.JavaMail.root@zimbra.anl.gov>
	<AANLkTikD61y+YN+QvQGqx+aZ=dNgAwXVoVff3zcLx3AK@mail.gmail.com>
Message-ID: <AANLkTimFWncMMwOWn5F2z-Z_ff3pvzk58EFPAEp4_HuX@mail.gmail.com>

Hey Sarah,

The extent of my (somewhat limited) knowledge on how to modify the swift
documentation is the wiki link that Mike gave. It should address the issues
I ran into this summer with character encoding and others. I have a copy of
docbook and fop in my home directory on bridled: /home/davidk/docbook-xsl
and /home/davidk/fop.

David

On Tue, Oct 5, 2010 at 4:08 PM, Sarah Kenny <skenny at uchicago.edu> wrote:

> does anyone know how/where it's currently maintained?
>
> ~sk
>
>
> On Tue, Oct 5, 2010 at 2:50 PM, Michael Wilde <wilde at mcs.anl.gov> wrote:
>
>> Justin has been looking for a way to:
>>
>> - have an online doc version per-release and per-branch
>>
>> - easily svn checkout the docs, test your changes to make sure they look
>> ready to post, commit, and push the changes to the right online version to
>> match that branch and release you are editing
>>
>> Sarah, can you coordinate with Justin on this, post the proposed method to
>> the list, and make it happen?
>>
>> Thanks,
>>
>> - Mike
>>
>>
>> ----- "Sarah Kenny" <skenny at uchicago.edu> wrote:
>>
>> > how can i edit the swift user guide? i'm seeing a few old bugs in
>> > bugzilla that could be cleared up with a very minor change to the
>> > doc...
>> >
>> > ~sk
>> >
>> > _______________________________________________
>> > Swift-devel mailing list
>> > Swift-devel at ci.uchicago.edu
>> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>>
>> --
>> Michael Wilde
>> Computation Institute, University of Chicago
>> Mathematics and Computer Science Division
>> Argonne National Laboratory
>>
>>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101005/78c8e01d/attachment.html>

From dk0966 at cs.ship.edu  Tue Oct  5 16:32:15 2010
From: dk0966 at cs.ship.edu (David Kelly)
Date: Tue, 5 Oct 2010 17:32:15 -0400
Subject: [Swift-devel] editing swift user guide
In-Reply-To: <AANLkTikD61y+YN+QvQGqx+aZ=dNgAwXVoVff3zcLx3AK@mail.gmail.com>
References: <AANLkTimH6rCmxfLisqXc+Vm_3MNrDzGQBU3+kmKrHJTb@mail.gmail.com>
	<1624213036.725411286308204739.JavaMail.root@zimbra.anl.gov>
	<AANLkTikD61y+YN+QvQGqx+aZ=dNgAwXVoVff3zcLx3AK@mail.gmail.com>
Message-ID: <AANLkTimFWncMMwOWn5F2z-Z_ff3pvzk58EFPAEp4_HuX@mail.gmail.com>

Hey Sarah,

The extent of my (somewhat limited) knowledge on how to modify the swift
documentation is the wiki link that Mike gave. It should address the issues
I ran into this summer with character encoding and others. I have a copy of
docbook and fop in my home directory on bridled: /home/davidk/docbook-xsl
and /home/davidk/fop.

David

On Tue, Oct 5, 2010 at 4:08 PM, Sarah Kenny <skenny at uchicago.edu> wrote:

> does anyone know how/where it's currently maintained?
>
> ~sk
>
>
> On Tue, Oct 5, 2010 at 2:50 PM, Michael Wilde <wilde at mcs.anl.gov> wrote:
>
>> Justin has been looking for a way to:
>>
>> - have an online doc version per-release and per-branch
>>
>> - easily svn checkout the docs, test your changes to make sure they look
>> ready to post, commit, and push the changes to the right online version to
>> match that branch and release you are editing
>>
>> Sarah, can you coordinate with Justin on this, post the proposed method to
>> the list, and make it happen?
>>
>> Thanks,
>>
>> - Mike
>>
>>
>> ----- "Sarah Kenny" <skenny at uchicago.edu> wrote:
>>
>> > how can i edit the swift user guide? i'm seeing a few old bugs in
>> > bugzilla that could be cleared up with a very minor change to the
>> > doc...
>> >
>> > ~sk
>> >
>> > _______________________________________________
>> > Swift-devel mailing list
>> > Swift-devel at ci.uchicago.edu
>> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>>
>> --
>> Michael Wilde
>> Computation Institute, University of Chicago
>> Mathematics and Computer Science Division
>> Argonne National Laboratory
>>
>>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101005/78c8e01d/attachment-0001.html>

From skenny at uchicago.edu  Tue Oct  5 18:11:17 2010
From: skenny at uchicago.edu (Sarah Kenny)
Date: Tue, 5 Oct 2010 18:11:17 -0500
Subject: [Swift-devel] new commits
In-Reply-To: <1286313183.16770.1.camel@blabla2.none>
References: <AANLkTimPsYeivXNZzj92ArWcPi1onU2C9D6ReVT28Z_J@mail.gmail.com>
	<1286301513.15904.2.camel@blabla2.none>
	<AANLkTimcKpH-4fKA8DM_kARAGybph+wsEbkyRtuvwe7m@mail.gmail.com>
	<1286313183.16770.1.camel@blabla2.none>
Message-ID: <AANLkTimOKqZH1Z3R=gb=vA9V4w2Eycu=gNz03iYHv+a2@mail.gmail.com>

On Tue, Oct 5, 2010 at 4:13 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> On Tue, 2010-10-05 at 14:31 -0500, Sarah Kenny wrote:
> > On Tue, Oct 5, 2010 at 12:58 PM, Mihael Hategan <hategan at mcs.anl.gov>
> > wrote:
> >         I think I committed some changes based on your patch. One
> >         problem was
> >         that gram2 doesn't seem to support a job name. Did you find a
> >         way around
> >         that?
> >
> > hmm, it is working on abe (gram2) and queenbee (gram5)...were you
> > getting an error running it elsewhere or did it just not pick up the
> > name? i'm thinking if it works for some gram versions and not others
> > it might be ok to commit unless it actually causes an error in
> > versions where the name is not passed (?)
>
> That was the problem. It caused an error.
>

ok...do you know where you ran it that you were able to produce the error?



> >
> > i thought you had committed (or were going to commit) the change as
> > well, but i'm not seeing it here:
> >
> >
> http://cogkit.svn.sourceforge.net/viewvc/cogkit/branches/4.1.7/src/cog/modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/Settings.java?revision=2623&view=markup&pathrev=2623
>
> It's in trunk.
>
> But:
>
> http://cogkit.svn.sourceforge.net/viewvc/cogkit/trunk/current/src/cog/modules/provider-gt2/src/org/globus/cog/abstraction/impl/execution/gt2/JobSubmissionTaskHandler.java?r1=2815&r2=2867
>
> >
> > is there somewhere else i should be looking?
> >
> >
> >
> >
> >         On Tue, 2010-10-05 at 12:26 -0500, Sarah Kenny wrote:
> >         > hi all, i have a fix to swift that allows the jobName
> >         parameter to be
> >         > passed to GRAM (so the job name actually shows up in the
> >         queue). this
> >         > will be my first commit to the cog/swift source so i want to
> >         know
> >         > where the best place is to do this? i'd like it to be in the
> >         stable
> >         > release so that it's there when we update. my user group
> >         (HNL) has
> >         > been using the stable branch with this fix added for several
> >         months so
> >         > it's pretty well-tested.
> >         >
> >         > i guess the question is would it be best to commit this to
> >         the
> >         > development code (which as i understand is
> >         cogkit/trunk/current) to be
> >         > migrated to the stable code (cogkit/branches/4.1.7) in the
> >         near future
> >         > when we decide to update the stable branch?
> >         >
> >         > also, i'm guessing i'll need to get access (from mihael or
> >         mike
> >         > wilde?) to the cog source...
> >
> >
> >         I can do that. Do you have a sourceforge account?
> >
> >
> >
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101005/2db249f4/attachment.html>

From skenny at uchicago.edu  Tue Oct  5 18:15:17 2010
From: skenny at uchicago.edu (Sarah Kenny)
Date: Tue, 5 Oct 2010 18:15:17 -0500
Subject: [Swift-devel] new commits
In-Reply-To: <AANLkTimcKpH-4fKA8DM_kARAGybph+wsEbkyRtuvwe7m@mail.gmail.com>
References: <AANLkTimPsYeivXNZzj92ArWcPi1onU2C9D6ReVT28Z_J@mail.gmail.com>
	<1286301513.15904.2.camel@blabla2.none>
	<AANLkTimcKpH-4fKA8DM_kARAGybph+wsEbkyRtuvwe7m@mail.gmail.com>
Message-ID: <AANLkTikZ15Zvt1--EjLR9+pkB_wpxMQBn98YT9FwaXTJ@mail.gmail.com>

On Tue, Oct 5, 2010 at 2:31 PM, Sarah Kenny <skenny at uchicago.edu> wrote:

> On Tue, Oct 5, 2010 at 12:58 PM, Mihael Hategan <hategan at mcs.anl.gov>wrote:
>
>> I think I committed some changes based on your patch. One problem was
>> that gram2 doesn't seem to support a job name. Did you find a way around
>> that?
>>
>
> hmm, it is working on abe (gram2) and queenbee (gram5)...were you getting
> an error running it elsewhere or did it just not pick up the name? i'm
> thinking if it works for some gram versions and not others it might be ok to
> commit unless it actually causes an error in versions where the name is not
> passed (?)
>
> i thought you had committed (or were going to commit) the change as well,
> but i'm not seeing it here:
>
>
> http://cogkit.svn.sourceforge.net/viewvc/cogkit/branches/4.1.7/src/cog/modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/Settings.java?revision=2623&view=markup&pathrev=2623
>
> is there somewhere else i should be looking?
>
>
>>
>>
>> On Tue, 2010-10-05 at 12:26 -0500, Sarah Kenny wrote:
>> > hi all, i have a fix to swift that allows the jobName parameter to be
>> > passed to GRAM (so the job name actually shows up in the queue). this
>> > will be my first commit to the cog/swift source so i want to know
>> > where the best place is to do this? i'd like it to be in the stable
>> > release so that it's there when we update. my user group (HNL) has
>> > been using the stable branch with this fix added for several months so
>> > it's pretty well-tested.
>> >
>> > i guess the question is would it be best to commit this to the
>> > development code (which as i understand is cogkit/trunk/current) to be
>> > migrated to the stable code (cogkit/branches/4.1.7) in the near future
>> > when we decide to update the stable branch?
>> >
>> > also, i'm guessing i'll need to get access (from mihael or mike
>> > wilde?) to the cog source...
>>
>> I can do that. Do you have a sourceforge account?
>>
>
yes, user name is skenny24
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101005/c975db28/attachment.html>

From skenny at uchicago.edu  Tue Oct  5 19:40:49 2010
From: skenny at uchicago.edu (Sarah Kenny)
Date: Tue, 5 Oct 2010 19:40:49 -0500
Subject: [Swift-devel] swift versions
Message-ID: <AANLkTinGjCYiwN6bRnWXO5mOMPB0B9MEgQJL7mBwfVTB@mail.gmail.com>

ok, just want to make sure i fully understand the state of the code:

these are the most current/stable versions of swift & cog (that which we
tell users to download):

swift: branch/1.0 (https://trac.ci.uchicago.edu/swift/browser/branches/1.0)
cog: branches/4.1.7 (
http://cogkit.svn.sourceforge.net/viewvc/cogkit/branches/4.1.7/)

these are the development (less stable) versions of the code that we should
be committing changes to:

swift: trunk/ (https://trac.ci.uchicago.edu/swift/browser/trunk)
cog: trunk/current (
http://cogkit.svn.sourceforge.net/viewvc/cogkit/trunk/current/)

assuming that's correct...

for swift, what are the other/multiple directories under branches/ that do
not appear to be releases or release candidates?
is the intent, upon the next swift release (whenever that may be) to update
swift branch/1.0 with the changes that have been committed to trunk/ ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101005/f57cbcb0/attachment.html>

From bugzilla-daemon at mcs.anl.gov  Tue Oct  5 20:27:30 2010
From: bugzilla-daemon at mcs.anl.gov (bugzilla-daemon at mcs.anl.gov)
Date: Tue,  5 Oct 2010 20:27:30 -0500 (CDT)
Subject: [Swift-devel] [Bug 100] deprecate workflow word?
In-Reply-To: <bug-100-21@http.bugzilla.mcs.anl.gov/swift/>
References: <bug-100-21@http.bugzilla.mcs.anl.gov/swift/>
Message-ID: <20101006012730.4323A2CD6C@wind.mcs.anl.gov>

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=100


skenny <skenny at uchicago.edu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |skenny at uchicago.edu
         Resolution|                            |WONTFIX




-- 
Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.


From bugzilla-daemon at mcs.anl.gov  Tue Oct  5 20:36:42 2010
From: bugzilla-daemon at mcs.anl.gov (bugzilla-daemon at mcs.anl.gov)
Date: Tue,  5 Oct 2010 20:36:42 -0500 (CDT)
Subject: [Swift-devel] [Bug 153] SGE adapter for gram acting weird on
	TACC_Ranger
In-Reply-To: <bug-153-21@http.bugzilla.mcs.anl.gov/swift/>
References: <bug-153-21@http.bugzilla.mcs.anl.gov/swift/>
Message-ID: <20101006013642.923B42CD79@wind.mcs.anl.gov>

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=153


skenny <skenny at uchicago.edu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |WORKSFORME




-- 
Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.


From bugzilla-daemon at mcs.anl.gov  Tue Oct  5 20:45:42 2010
From: bugzilla-daemon at mcs.anl.gov (bugzilla-daemon at mcs.anl.gov)
Date: Tue,  5 Oct 2010 20:45:42 -0500 (CDT)
Subject: [Swift-devel] [Bug 163] ext mapper doesn't like being used for
	input files.
In-Reply-To: <bug-163-21@http.bugzilla.mcs.anl.gov/swift/>
References: <bug-163-21@http.bugzilla.mcs.anl.gov/swift/>
Message-ID: <20101006014542.E738D2CBAF@wind.mcs.anl.gov>

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=163


skenny <skenny at uchicago.edu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |skenny at uchicago.edu
         Resolution|                            |WORKSFORME




--- Comment #3 from skenny <skenny at uchicago.edu>  2010-10-05 20:45:42 ---
ambiguity of error message will be submitted as a different ticket.

-- 
Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching the reporter.


From skenny at uchicago.edu  Tue Oct  5 20:52:45 2010
From: skenny at uchicago.edu (Sarah Kenny)
Date: Tue, 5 Oct 2010 20:52:45 -0500
Subject: [Swift-devel] clustering deprecated?
Message-ID: <AANLkTikiEf3zQGavuVtk5MXk4iR7h7Vm7uLeXz8NuiCU@mail.gmail.com>

my understanding is that clustering in swift is no-longer used and has been
completely replaced by coasters. is that true or is this feature still
used/functional? that is, are we intending to actually fix bugs related to
clustering?

~sk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101005/6db0c23e/attachment.html>

From hategan at mcs.anl.gov  Tue Oct  5 21:16:58 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 05 Oct 2010 19:16:58 -0700
Subject: [Swift-devel] new commits
In-Reply-To: <AANLkTimOKqZH1Z3R=gb=vA9V4w2Eycu=gNz03iYHv+a2@mail.gmail.com>
References: <AANLkTimPsYeivXNZzj92ArWcPi1onU2C9D6ReVT28Z_J@mail.gmail.com>
	<1286301513.15904.2.camel@blabla2.none>
	<AANLkTimcKpH-4fKA8DM_kARAGybph+wsEbkyRtuvwe7m@mail.gmail.com>
	<1286313183.16770.1.camel@blabla2.none>
	<AANLkTimOKqZH1Z3R=gb=vA9V4w2Eycu=gNz03iYHv+a2@mail.gmail.com>
Message-ID: <1286331418.19448.2.camel@blabla2.none>

On Tue, 2010-10-05 at 18:11 -0500, Sarah Kenny wrote:
> 
> 
> On Tue, Oct 5, 2010 at 4:13 PM, Mihael Hategan <hategan at mcs.anl.gov>
> wrote:
[...]
>         >         I think I committed some changes based on your
>         patch. One
>         >         problem was
>         >         that gram2 doesn't seem to support a job name. Did
>         you find a
>         >         way around
>         >         that?
>         >
>         > hmm, it is working on abe (gram2) and queenbee
>         (gram5)...were you
>         > getting an error running it elsewhere or did it just not
>         pick up the
>         > name? i'm thinking if it works for some gram versions and
>         not others
>         > it might be ok to commit unless it actually causes an error
>         in
>         > versions where the name is not passed (?)
>         
>         
>         That was the problem. It caused an error.
> 
> ok...do you know where you ran it that you were able to produce the
> error?
> 
http://mail.ci.uchicago.edu/mailman/private/swift-devel/2010-August/006730.html

You may need a password for that archive, which I will disclose through
some other means. Otherwise subject was "gram2 coaster workers
terminating prematurely"

[...]




From hategan at mcs.anl.gov  Tue Oct  5 21:20:13 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 05 Oct 2010 19:20:13 -0700
Subject: [Swift-devel] new commits
In-Reply-To: <AANLkTikZ15Zvt1--EjLR9+pkB_wpxMQBn98YT9FwaXTJ@mail.gmail.com>
References: <AANLkTimPsYeivXNZzj92ArWcPi1onU2C9D6ReVT28Z_J@mail.gmail.com>
	<1286301513.15904.2.camel@blabla2.none>
	<AANLkTimcKpH-4fKA8DM_kARAGybph+wsEbkyRtuvwe7m@mail.gmail.com>
	<AANLkTikZ15Zvt1--EjLR9+pkB_wpxMQBn98YT9FwaXTJ@mail.gmail.com>
Message-ID: <1286331613.19448.4.camel@blabla2.none>

On Tue, 2010-10-05 at 18:15 -0500, Sarah Kenny wrote:

>                 >
>                 > also, i'm guessing i'll need to get access (from
>                 mihael or mike
>                 > wilde?) to the cog source...
>                 
>                 
>                 I can do that. Do you have a sourceforge account?
> 
> yes, user name is skenny24
>  
> 
You are in.



From hategan at mcs.anl.gov  Tue Oct  5 21:22:33 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 05 Oct 2010 19:22:33 -0700
Subject: [Swift-devel] swift versions
In-Reply-To: <AANLkTinGjCYiwN6bRnWXO5mOMPB0B9MEgQJL7mBwfVTB@mail.gmail.com>
References: <AANLkTinGjCYiwN6bRnWXO5mOMPB0B9MEgQJL7mBwfVTB@mail.gmail.com>
Message-ID: <1286331753.19448.6.camel@blabla2.none>

On Tue, 2010-10-05 at 19:40 -0500, Sarah Kenny wrote:
> ok, just want to make sure i fully understand the state of the code:
> 
> these are the most current/stable versions of swift & cog (that which
> we tell users to download):
> 
> swift: branch/1.0
> (https://trac.ci.uchicago.edu/swift/browser/branches/1.0)
> cog: branches/4.1.7
> (http://cogkit.svn.sourceforge.net/viewvc/cogkit/branches/4.1.7/)
> 
> these are the development (less stable) versions of the code that we
> should be committing changes to:
> 
> swift: trunk/ (https://trac.ci.uchicago.edu/swift/browser/trunk)
> cog: trunk/current
> (http://cogkit.svn.sourceforge.net/viewvc/cogkit/trunk/current/)
> 
> assuming that's correct...
> 
> for swift, what are the other/multiple directories under branches/
> that do not appear to be releases or release candidates?
> is the intent, upon the next swift release (whenever that may be) to
> update swift branch/1.0 with the changes that have been committed to
> trunk/ ? 

Right. There are two reasons for doing branches:
1. if some work divergent from trunk is needed
2. to stabilize for a release

For (1), if thinks work out OK, you eventually merge them back into
trunk.

(2)s also get merged back into trunk in one way or another, since they
may contain bug fixes.



From hategan at mcs.anl.gov  Tue Oct  5 21:24:06 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 05 Oct 2010 19:24:06 -0700
Subject: [Swift-devel] clustering deprecated?
In-Reply-To: <AANLkTikiEf3zQGavuVtk5MXk4iR7h7Vm7uLeXz8NuiCU@mail.gmail.com>
References: <AANLkTikiEf3zQGavuVtk5MXk4iR7h7Vm7uLeXz8NuiCU@mail.gmail.com>
Message-ID: <1286331846.19448.8.camel@blabla2.none>

On Tue, 2010-10-05 at 20:52 -0500, Sarah Kenny wrote:
> my understanding is that clustering in swift is no-longer used and has
> been completely replaced by coasters. is that true or is this feature
> still used/functional? that is, are we intending to actually fix bugs
> related to clustering?

It depends on whether we want to keep it around, just in case. My vote
is no because that means less clutter and I haven't heard of it being
used in a long while. Vote?



From wilde at mcs.anl.gov  Tue Oct  5 22:41:19 2010
From: wilde at mcs.anl.gov (wilde at mcs.anl.gov)
Date: Tue, 5 Oct 2010 21:41:19 -0600 (GMT-06:00)
Subject: [Swift-devel] clustering deprecated?
In-Reply-To: <165456407.743911286336409256.JavaMail.root@zimbra.anl.gov>
Message-ID: <482926478.743931286336479451.JavaMail.root@zimbra.anl.gov>

I'd be hesitant to remove it just yet; lets see where it may be causing a nuisance first. Also removing it has some risks until we have a more solid testing suite.

Maybe 6 months from now?

- Mike

----- "Mihael Hategan" <hategan at mcs.anl.gov> wrote:

> On Tue, 2010-10-05 at 20:52 -0500, Sarah Kenny wrote:
> > my understanding is that clustering in swift is no-longer used and
> has
> > been completely replaced by coasters. is that true or is this
> feature
> > still used/functional? that is, are we intending to actually fix
> bugs
> > related to clustering?
> 
> It depends on whether we want to keep it around, just in case. My
> vote
> is no because that means less clutter and I haven't heard of it being
> used in a long while. Vote?
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From hategan at mcs.anl.gov  Tue Oct  5 22:42:20 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 05 Oct 2010 20:42:20 -0700
Subject: [Swift-devel] clustering deprecated?
In-Reply-To: <482926478.743931286336479451.JavaMail.root@zimbra.anl.gov>
References: <482926478.743931286336479451.JavaMail.root@zimbra.anl.gov>
Message-ID: <1286336540.20599.1.camel@blabla2.none>

On Tue, 2010-10-05 at 21:41 -0600, wilde at mcs.anl.gov wrote:
> I'd be hesitant to remove it just yet; lets see where it may be
> causing a nuisance first. Also removing it has some risks until we
> have a more solid testing suite.

Presumably we'd remove it from trunk and test that before moving it into
a release branch.

> 
> Maybe 6 months from now?

At our current delta t of trunk to release, 6 months in a release
probably means removing it from trunk now.

Mihael



From wozniak at mcs.anl.gov  Wed Oct  6 09:29:20 2010
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Wed, 6 Oct 2010 09:29:20 -0500 (CDT)
Subject: [Swift-devel] clustering deprecated?
In-Reply-To: <1286336540.20599.1.camel@blabla2.none>
References: <482926478.743931286336479451.JavaMail.root@zimbra.anl.gov>
	<1286336540.20599.1.camel@blabla2.none>
Message-ID: <alpine.DEB.2.00.1010060928270.2548@wozniak-desktop.mcs.anl.gov>

On Tue, 5 Oct 2010, Mihael Hategan wrote:

> On Tue, 2010-10-05 at 21:41 -0600, wilde at mcs.anl.gov wrote:
>> I'd be hesitant to remove it just yet; lets see where it may be
>> causing a nuisance first. Also removing it has some risks until we
>> have a more solid testing suite.
>
> Presumably we'd remove it from trunk and test that before moving it into
> a release branch.
>
>>
>> Maybe 6 months from now?
>
> At our current delta t of trunk to release, 6 months in a release
> probably means removing it from trunk now.
>
> Mihael

Maybe we should post this on swift-users?

-- 
Justin M Wozniak


From wilde at mcs.anl.gov  Wed Oct  6 14:33:43 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 6 Oct 2010 13:33:43 -0600 (GMT-06:00)
Subject: [Swift-devel] Making Swift arrays take string indices
Message-ID: <2119195391.783801286393623509.JavaMail.root@zimbra.anl.gov>

Mihael, back when you did the Swift internals classes we talked about the feasibility of making this change, and you suggested it might be a good student project.

Several students from a CS class in Sri Lanka are interested in Swift projects.

Can you post some suggestions to this list on how to go about this enhancements? (ie what code to look at..., what tests to try...)

Thanks,

Mike



From wozniak at mcs.anl.gov  Thu Oct  7 11:04:18 2010
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Thu, 7 Oct 2010 11:04:18 -0500 (Central Daylight Time)
Subject: [Swift-devel] swift versions
In-Reply-To: <1286331753.19448.6.camel@blabla2.none>
References: <AANLkTinGjCYiwN6bRnWXO5mOMPB0B9MEgQJL7mBwfVTB@mail.gmail.com>
	<1286331753.19448.6.camel@blabla2.none>
Message-ID: <alpine.WNT.2.00.1010071102160.3252@justinwozniak>

On Tue, 5 Oct 2010, Mihael Hategan wrote:

> On Tue, 2010-10-05 at 19:40 -0500, Sarah Kenny wrote:
>> ok, just want to make sure i fully understand the state of the code:
>>
>> these are the most current/stable versions of swift & cog (that which
>> we tell users to download):
>>
>> swift: branch/1.0
>> (https://trac.ci.uchicago.edu/swift/browser/branches/1.0)
>> cog: branches/4.1.7
>> (http://cogkit.svn.sourceforge.net/viewvc/cogkit/branches/4.1.7/)
>>
>> these are the development (less stable) versions of the code that we
>> should be committing changes to:
>>
>> swift: trunk/ (https://trac.ci.uchicago.edu/swift/browser/trunk)
>> cog: trunk/current
>> (http://cogkit.svn.sourceforge.net/viewvc/cogkit/trunk/current/)
>>
>> assuming that's correct...
>>
>> for swift, what are the other/multiple directories under branches/
>> that do not appear to be releases or release candidates?
>> is the intent, upon the next swift release (whenever that may be) to
>> update swift branch/1.0 with the changes that have been committed to
>> trunk/ ?
>
> Right. There are two reasons for doing branches:
> 1. if some work divergent from trunk is needed
> 2. to stabilize for a release
>
> For (1), if thinks work out OK, you eventually merge them back into
> trunk.
>
> (2)s also get merged back into trunk in one way or another, since they
> may contain bug fixes.
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel

So for Swift 1.1, can we have a CoG 4.1.8?

-- 
Justin M Wozniak


From skenny at uchicago.edu  Thu Oct  7 11:56:21 2010
From: skenny at uchicago.edu (Sarah Kenny)
Date: Thu, 7 Oct 2010 11:56:21 -0500
Subject: [Swift-devel] swift versions
In-Reply-To: <alpine.WNT.2.00.1010071102160.3252@justinwozniak>
References: <AANLkTinGjCYiwN6bRnWXO5mOMPB0B9MEgQJL7mBwfVTB@mail.gmail.com>
	<1286331753.19448.6.camel@blabla2.none>
	<alpine.WNT.2.00.1010071102160.3252@justinwozniak>
Message-ID: <AANLkTimMQuWc7DUSLf=EgsZWU=FvLsC-nhd154z4PvyF@mail.gmail.com>

does the 6 month release time that mihael alluded to seem doable? that is,
do you think we should shoot for the next release being around the 1st week
of april 2011?...during which time we would phase out clustering (provided
we don't get anyone on the user list saying they need it...which i sincerely
doubt).

~sk


On Thu, Oct 7, 2010 at 11:04 AM, Justin M Wozniak <wozniak at mcs.anl.gov>wrote:

> On Tue, 5 Oct 2010, Mihael Hategan wrote:
>
>  On Tue, 2010-10-05 at 19:40 -0500, Sarah Kenny wrote:
>>
>>> ok, just want to make sure i fully understand the state of the code:
>>>
>>> these are the most current/stable versions of swift & cog (that which
>>> we tell users to download):
>>>
>>> swift: branch/1.0
>>> (https://trac.ci.uchicago.edu/swift/browser/branches/1.0)
>>> cog: branches/4.1.7
>>> (http://cogkit.svn.sourceforge.net/viewvc/cogkit/branches/4.1.7/)
>>>
>>> these are the development (less stable) versions of the code that we
>>> should be committing changes to:
>>>
>>> swift: trunk/ (https://trac.ci.uchicago.edu/swift/browser/trunk)
>>> cog: trunk/current
>>> (http://cogkit.svn.sourceforge.net/viewvc/cogkit/trunk/current/)
>>>
>>> assuming that's correct...
>>>
>>> for swift, what are the other/multiple directories under branches/
>>> that do not appear to be releases or release candidates?
>>> is the intent, upon the next swift release (whenever that may be) to
>>> update swift branch/1.0 with the changes that have been committed to
>>> trunk/ ?
>>>
>>
>> Right. There are two reasons for doing branches:
>> 1. if some work divergent from trunk is needed
>> 2. to stabilize for a release
>>
>> For (1), if thinks work out OK, you eventually merge them back into
>> trunk.
>>
>> (2)s also get merged back into trunk in one way or another, since they
>> may contain bug fixes.
>>
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>>
>
> So for Swift 1.1, can we have a CoG 4.1.8?
>
> --
> Justin M Wozniak
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101007/ad1ec4d6/attachment.html>

From wozniak at mcs.anl.gov  Thu Oct  7 12:51:32 2010
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Thu, 7 Oct 2010 12:51:32 -0500 (CDT)
Subject: [Swift-devel] swift versions
In-Reply-To: <AANLkTimMQuWc7DUSLf=EgsZWU=FvLsC-nhd154z4PvyF@mail.gmail.com>
References: <AANLkTinGjCYiwN6bRnWXO5mOMPB0B9MEgQJL7mBwfVTB@mail.gmail.com>
	<1286331753.19448.6.camel@blabla2.none>
	<alpine.WNT.2.00.1010071102160.3252@justinwozniak>
	<AANLkTimMQuWc7DUSLf=EgsZWU=FvLsC-nhd154z4PvyF@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1010071250240.1793@wozniak-desktop.mcs.anl.gov>


Yeah, I would like to get the branches (and branched manuals) formed on 
the short term.

On Thu, 7 Oct 2010, Sarah Kenny wrote:

> does the 6 month release time that mihael alluded to seem doable? that is,
> do you think we should shoot for the next release being around the 1st week
> of april 2011?...during which time we would phase out clustering (provided
> we don't get anyone on the user list saying they need it...which i sincerely
> doubt).
>
> ~sk
>
>
> On Thu, Oct 7, 2010 at 11:04 AM, Justin M Wozniak <wozniak at mcs.anl.gov>wrote:
>
>> On Tue, 5 Oct 2010, Mihael Hategan wrote:
>>
>>  On Tue, 2010-10-05 at 19:40 -0500, Sarah Kenny wrote:
>>>
>>>> ok, just want to make sure i fully understand the state of the code:
>>>>
>>>> these are the most current/stable versions of swift & cog (that which
>>>> we tell users to download):
>>>>
>>>> swift: branch/1.0
>>>> (https://trac.ci.uchicago.edu/swift/browser/branches/1.0)
>>>> cog: branches/4.1.7
>>>> (http://cogkit.svn.sourceforge.net/viewvc/cogkit/branches/4.1.7/)
>>>>
>>>> these are the development (less stable) versions of the code that we
>>>> should be committing changes to:
>>>>
>>>> swift: trunk/ (https://trac.ci.uchicago.edu/swift/browser/trunk)
>>>> cog: trunk/current
>>>> (http://cogkit.svn.sourceforge.net/viewvc/cogkit/trunk/current/)
>>>>
>>>> assuming that's correct...
>>>>
>>>> for swift, what are the other/multiple directories under branches/
>>>> that do not appear to be releases or release candidates?
>>>> is the intent, upon the next swift release (whenever that may be) to
>>>> update swift branch/1.0 with the changes that have been committed to
>>>> trunk/ ?
>>>>
>>>
>>> Right. There are two reasons for doing branches:
>>> 1. if some work divergent from trunk is needed
>>> 2. to stabilize for a release
>>>
>>> For (1), if thinks work out OK, you eventually merge them back into
>>> trunk.
>>>
>>> (2)s also get merged back into trunk in one way or another, since they
>>> may contain bug fixes.
>>>
>>> _______________________________________________
>>> Swift-devel mailing list
>>> Swift-devel at ci.uchicago.edu
>>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>>>
>>
>> So for Swift 1.1, can we have a CoG 4.1.8?
>>
>> --
>> Justin M Wozniak
>>
>

-- 
Justin M Wozniak


From hategan at mcs.anl.gov  Thu Oct  7 13:08:28 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 07 Oct 2010 11:08:28 -0700
Subject: [Swift-devel] swift versions
In-Reply-To: <alpine.DEB.2.00.1010071250240.1793@wozniak-desktop.mcs.anl.gov>
References: <AANLkTinGjCYiwN6bRnWXO5mOMPB0B9MEgQJL7mBwfVTB@mail.gmail.com>
	<1286331753.19448.6.camel@blabla2.none>
	<alpine.WNT.2.00.1010071102160.3252@justinwozniak>
	<AANLkTimMQuWc7DUSLf=EgsZWU=FvLsC-nhd154z4PvyF@mail.gmail.com>
	<alpine.DEB.2.00.1010071250240.1793@wozniak-desktop.mcs.anl.gov>
Message-ID: <1286474908.25937.1.camel@blabla2.none>

Right. I think 1.0/4.1.7 should go out soon. And then 1.1/4.1.8 next. 6
months was just a guess at how much it will take us to get to 1.1 and is
by no means what I think should happen. But my suspicion is that it will
take on the order of months.

On Thu, 2010-10-07 at 12:51 -0500, Justin M Wozniak wrote:
> Yeah, I would like to get the branches (and branched manuals) formed on 
> the short term.
> 
> On Thu, 7 Oct 2010, Sarah Kenny wrote:
> 
> > does the 6 month release time that mihael alluded to seem doable? that is,
> > do you think we should shoot for the next release being around the 1st week
> > of april 2011?...during which time we would phase out clustering (provided
> > we don't get anyone on the user list saying they need it...which i sincerely
> > doubt).
> >
> > ~sk
> >
> >
> > On Thu, Oct 7, 2010 at 11:04 AM, Justin M Wozniak <wozniak at mcs.anl.gov>wrote:
> >
> >> On Tue, 5 Oct 2010, Mihael Hategan wrote:
> >>
> >>  On Tue, 2010-10-05 at 19:40 -0500, Sarah Kenny wrote:
> >>>
> >>>> ok, just want to make sure i fully understand the state of the code:
> >>>>
> >>>> these are the most current/stable versions of swift & cog (that which
> >>>> we tell users to download):
> >>>>
> >>>> swift: branch/1.0
> >>>> (https://trac.ci.uchicago.edu/swift/browser/branches/1.0)
> >>>> cog: branches/4.1.7
> >>>> (http://cogkit.svn.sourceforge.net/viewvc/cogkit/branches/4.1.7/)
> >>>>
> >>>> these are the development (less stable) versions of the code that we
> >>>> should be committing changes to:
> >>>>
> >>>> swift: trunk/ (https://trac.ci.uchicago.edu/swift/browser/trunk)
> >>>> cog: trunk/current
> >>>> (http://cogkit.svn.sourceforge.net/viewvc/cogkit/trunk/current/)
> >>>>
> >>>> assuming that's correct...
> >>>>
> >>>> for swift, what are the other/multiple directories under branches/
> >>>> that do not appear to be releases or release candidates?
> >>>> is the intent, upon the next swift release (whenever that may be) to
> >>>> update swift branch/1.0 with the changes that have been committed to
> >>>> trunk/ ?
> >>>>
> >>>
> >>> Right. There are two reasons for doing branches:
> >>> 1. if some work divergent from trunk is needed
> >>> 2. to stabilize for a release
> >>>
> >>> For (1), if thinks work out OK, you eventually merge them back into
> >>> trunk.
> >>>
> >>> (2)s also get merged back into trunk in one way or another, since they
> >>> may contain bug fixes.
> >>>
> >>> _______________________________________________
> >>> Swift-devel mailing list
> >>> Swift-devel at ci.uchicago.edu
> >>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> >>>
> >>
> >> So for Swift 1.1, can we have a CoG 4.1.8?
> >>
> >> --
> >> Justin M Wozniak
> >>
> >
> 




From skenny at uchicago.edu  Thu Oct  7 14:16:42 2010
From: skenny at uchicago.edu (Sarah Kenny)
Date: Thu, 7 Oct 2010 14:16:42 -0500
Subject: [Swift-devel] swift versions
In-Reply-To: <1286474908.25937.1.camel@blabla2.none>
References: <AANLkTinGjCYiwN6bRnWXO5mOMPB0B9MEgQJL7mBwfVTB@mail.gmail.com>
	<1286331753.19448.6.camel@blabla2.none>
	<alpine.WNT.2.00.1010071102160.3252@justinwozniak>
	<AANLkTimMQuWc7DUSLf=EgsZWU=FvLsC-nhd154z4PvyF@mail.gmail.com>
	<alpine.DEB.2.00.1010071250240.1793@wozniak-desktop.mcs.anl.gov>
	<1286474908.25937.1.camel@blabla2.none>
Message-ID: <AANLkTimGXuDPPbXoDDupsnWMaTREdBEuJ9rugVWcRrDQ@mail.gmail.com>

On Thu, Oct 7, 2010 at 1:08 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> Right. I think 1.0/4.1.7 should go out soon.


ok, so i guess we should decide what 'soon' means ;) i am currently going
thru the old bugs in bugzilla (at least trying to close out things that have
been already fixed or are no-longer applicable, etc), but perhaps it would
be good to determine if there are bigger issues outside of that that still
need to be dealt with before we can put what we've got into a stable release
and determine a time-frame...anything come to mind?

as far as documentation...does it make sense for each branch to have a full
copy of /ci/www/projects/swift under it which can then be merged with the
main/live copy whenever the code is merged? admittedly, i know nothing about
docbook, but from the standpoint of updating and merging this seems to make
sense to me (though feel free to suggest another way :)

~sk



> And then 1.1/4.1.8 next. 6
> months was just a guess at how much it will take us to get to 1.1 and is
> by no means what I think should happen. But my suspicion is that it will
> take on the order of months.
>
> On Thu, 2010-10-07 at 12:51 -0500, Justin M Wozniak wrote:
> > Yeah, I would like to get the branches (and branched manuals) formed on
> > the short term.
> >
> > On Thu, 7 Oct 2010, Sarah Kenny wrote:
> >
> > > does the 6 month release time that mihael alluded to seem doable? that
> is,
> > > do you think we should shoot for the next release being around the 1st
> week
> > > of april 2011?...during which time we would phase out clustering
> (provided
> > > we don't get anyone on the user list saying they need it...which i
> sincerely
> > > doubt).
> > >
> > > ~sk
> > >
> > >
> > > On Thu, Oct 7, 2010 at 11:04 AM, Justin M Wozniak <wozniak at mcs.anl.gov
> >wrote:
> > >
> > >> On Tue, 5 Oct 2010, Mihael Hategan wrote:
> > >>
> > >>  On Tue, 2010-10-05 at 19:40 -0500, Sarah Kenny wrote:
> > >>>
> > >>>> ok, just want to make sure i fully understand the state of the code:
> > >>>>
> > >>>> these are the most current/stable versions of swift & cog (that
> which
> > >>>> we tell users to download):
> > >>>>
> > >>>> swift: branch/1.0
> > >>>> (https://trac.ci.uchicago.edu/swift/browser/branches/1.0)
> > >>>> cog: branches/4.1.7
> > >>>> (http://cogkit.svn.sourceforge.net/viewvc/cogkit/branches/4.1.7/)
> > >>>>
> > >>>> these are the development (less stable) versions of the code that we
> > >>>> should be committing changes to:
> > >>>>
> > >>>> swift: trunk/ (https://trac.ci.uchicago.edu/swift/browser/trunk)
> > >>>> cog: trunk/current
> > >>>> (http://cogkit.svn.sourceforge.net/viewvc/cogkit/trunk/current/)
> > >>>>
> > >>>> assuming that's correct...
> > >>>>
> > >>>> for swift, what are the other/multiple directories under branches/
> > >>>> that do not appear to be releases or release candidates?
> > >>>> is the intent, upon the next swift release (whenever that may be) to
> > >>>> update swift branch/1.0 with the changes that have been committed to
> > >>>> trunk/ ?
> > >>>>
> > >>>
> > >>> Right. There are two reasons for doing branches:
> > >>> 1. if some work divergent from trunk is needed
> > >>> 2. to stabilize for a release
> > >>>
> > >>> For (1), if thinks work out OK, you eventually merge them back into
> > >>> trunk.
> > >>>
> > >>> (2)s also get merged back into trunk in one way or another, since
> they
> > >>> may contain bug fixes.
> > >>>
> > >>> _______________________________________________
> > >>> Swift-devel mailing list
> > >>> Swift-devel at ci.uchicago.edu
> > >>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> > >>>
> > >>
> > >> So for Swift 1.1, can we have a CoG 4.1.8?
> > >>
> > >> --
> > >> Justin M Wozniak
> > >>
> > >
> >
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101007/963b19c0/attachment.html>

From wozniak at mcs.anl.gov  Thu Oct  7 14:28:23 2010
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Thu, 7 Oct 2010 14:28:23 -0500 (CDT)
Subject: [Swift-devel] swift versions
In-Reply-To: <AANLkTimGXuDPPbXoDDupsnWMaTREdBEuJ9rugVWcRrDQ@mail.gmail.com>
References: <AANLkTinGjCYiwN6bRnWXO5mOMPB0B9MEgQJL7mBwfVTB@mail.gmail.com>
	<1286331753.19448.6.camel@blabla2.none>
	<alpine.WNT.2.00.1010071102160.3252@justinwozniak>
	<AANLkTimMQuWc7DUSLf=EgsZWU=FvLsC-nhd154z4PvyF@mail.gmail.com>
	<alpine.DEB.2.00.1010071250240.1793@wozniak-desktop.mcs.anl.gov>
	<1286474908.25937.1.camel@blabla2.none>
	<AANLkTimGXuDPPbXoDDupsnWMaTREdBEuJ9rugVWcRrDQ@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1010071419540.1793@wozniak-desktop.mcs.anl.gov>

On Thu, 7 Oct 2010, Sarah Kenny wrote:

> On Thu, Oct 7, 2010 at 1:08 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>
>> Right. I think 1.0/4.1.7 should go out soon.
>
> ok, so i guess we should decide what 'soon' means ;) i am currently going
> thru the old bugs in bugzilla (at least trying to close out things that have
> been already fixed or are no-longer applicable, etc), but perhaps it would
> be good to determine if there are bigger issues outside of that that still
> need to be dealt with before we can put what we've got into a stable release
> and determine a time-frame...anything come to mind?
>
> as far as documentation...does it make sense for each branch to have a full
> copy of /ci/www/projects/swift under it which can then be merged with the
> main/live copy whenever the code is merged? admittedly, i know nothing about
> docbook, but from the standpoint of updating and merging this seems to make
> sense to me (though feel free to suggest another way :)
>
> ~sk

I was looking at the update.sh script earlier today- I propose we have one 
web site but multiple docs/guides directories, all accessible from the 
docs/index.php page.  Each of these would be associated with a branch. 
So, similar to the existing "Historical" section but for "future" branches 
as well.  That would take a small modification to the update.sh script and 
manual modification of the docs/index.php page for each version number.

We may also want to have the feature changes (past and future version 
numbers) available on that page but I think those can be plain text. 
These could be pulled directly from SVN as well.

-- 
Justin M Wozniak


From hategan at mcs.anl.gov  Thu Oct  7 14:34:00 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 07 Oct 2010 12:34:00 -0700
Subject: [Swift-devel] swift versions
In-Reply-To: <alpine.DEB.2.00.1010071419540.1793@wozniak-desktop.mcs.anl.gov>
References: <AANLkTinGjCYiwN6bRnWXO5mOMPB0B9MEgQJL7mBwfVTB@mail.gmail.com>
	<1286331753.19448.6.camel@blabla2.none>
	<alpine.WNT.2.00.1010071102160.3252@justinwozniak>
	<AANLkTimMQuWc7DUSLf=EgsZWU=FvLsC-nhd154z4PvyF@mail.gmail.com>
	<alpine.DEB.2.00.1010071250240.1793@wozniak-desktop.mcs.anl.gov>
	<1286474908.25937.1.camel@blabla2.none>
	<AANLkTimGXuDPPbXoDDupsnWMaTREdBEuJ9rugVWcRrDQ@mail.gmail.com>
	<alpine.DEB.2.00.1010071419540.1793@wozniak-desktop.mcs.anl.gov>
Message-ID: <1286480040.26601.0.camel@blabla2.none>

On Thu, 2010-10-07 at 14:28 -0500, Justin M Wozniak wrote:
> On Thu, 7 Oct 2010, Sarah Kenny wrote:
> 
> > On Thu, Oct 7, 2010 at 1:08 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> >
> >> Right. I think 1.0/4.1.7 should go out soon.
> >
> > ok, so i guess we should decide what 'soon' means ;) i am currently going
> > thru the old bugs in bugzilla (at least trying to close out things that have
> > been already fixed or are no-longer applicable, etc), but perhaps it would
> > be good to determine if there are bigger issues outside of that that still
> > need to be dealt with before we can put what we've got into a stable release
> > and determine a time-frame...anything come to mind?
> >
> > as far as documentation...does it make sense for each branch to have a full
> > copy of /ci/www/projects/swift under it which can then be merged with the
> > main/live copy whenever the code is merged? admittedly, i know nothing about
> > docbook, but from the standpoint of updating and merging this seems to make
> > sense to me (though feel free to suggest another way :)
> >
> > ~sk
> 
> I was looking at the update.sh script earlier today- I propose we have one 
> web site but multiple docs/guides directories, all accessible from the 
> docs/index.php page.  Each of these would be associated with a branch. 
> So, similar to the existing "Historical" section but for "future" branches 
> as well.  That would take a small modification to the update.sh script and 
> manual modification of the docs/index.php page for each version number.
> 
> We may also want to have the feature changes (past and future version 
> numbers) available on that page but I think those can be plain text. 
> These could be pulled directly from SVN as well.
> 

I agree. I generally believe that documentation should be kept in sync
with releases (and I also think that the effort of doing so is minimal).



From skenny at uchicago.edu  Thu Oct  7 14:39:11 2010
From: skenny at uchicago.edu (Sarah Kenny)
Date: Thu, 7 Oct 2010 14:39:11 -0500
Subject: [Swift-devel] swift versions
In-Reply-To: <1286480040.26601.0.camel@blabla2.none>
References: <AANLkTinGjCYiwN6bRnWXO5mOMPB0B9MEgQJL7mBwfVTB@mail.gmail.com>
	<1286331753.19448.6.camel@blabla2.none>
	<alpine.WNT.2.00.1010071102160.3252@justinwozniak>
	<AANLkTimMQuWc7DUSLf=EgsZWU=FvLsC-nhd154z4PvyF@mail.gmail.com>
	<alpine.DEB.2.00.1010071250240.1793@wozniak-desktop.mcs.anl.gov>
	<1286474908.25937.1.camel@blabla2.none>
	<AANLkTimGXuDPPbXoDDupsnWMaTREdBEuJ9rugVWcRrDQ@mail.gmail.com>
	<alpine.DEB.2.00.1010071419540.1793@wozniak-desktop.mcs.anl.gov>
	<1286480040.26601.0.camel@blabla2.none>
Message-ID: <AANLkTi=Z3QH0uSJrVN0d-FuBh5AnDCX_uoOqvbWkkp86@mail.gmail.com>

so, in this scenario, the changes to the doc that exist in each branch are
pushed to the main user doc when we do the release or am i missing a step
here?

On Thu, Oct 7, 2010 at 2:34 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> On Thu, 2010-10-07 at 14:28 -0500, Justin M Wozniak wrote:
> > On Thu, 7 Oct 2010, Sarah Kenny wrote:
> >
> > > On Thu, Oct 7, 2010 at 1:08 PM, Mihael Hategan <hategan at mcs.anl.gov>
> wrote:
> > >
> > >> Right. I think 1.0/4.1.7 should go out soon.
> > >
> > > ok, so i guess we should decide what 'soon' means ;) i am currently
> going
> > > thru the old bugs in bugzilla (at least trying to close out things that
> have
> > > been already fixed or are no-longer applicable, etc), but perhaps it
> would
> > > be good to determine if there are bigger issues outside of that that
> still
> > > need to be dealt with before we can put what we've got into a stable
> release
> > > and determine a time-frame...anything come to mind?
> > >
> > > as far as documentation...does it make sense for each branch to have a
> full
> > > copy of /ci/www/projects/swift under it which can then be merged with
> the
> > > main/live copy whenever the code is merged? admittedly, i know nothing
> about
> > > docbook, but from the standpoint of updating and merging this seems to
> make
> > > sense to me (though feel free to suggest another way :)
> > >
> > > ~sk
> >
> > I was looking at the update.sh script earlier today- I propose we have
> one
> > web site but multiple docs/guides directories, all accessible from the
> > docs/index.php page.  Each of these would be associated with a branch.
> > So, similar to the existing "Historical" section but for "future"
> branches
> > as well.  That would take a small modification to the update.sh script
> and
> > manual modification of the docs/index.php page for each version number.
> >
> > We may also want to have the feature changes (past and future version
> > numbers) available on that page but I think those can be plain text.
> > These could be pulled directly from SVN as well.
> >
>
> I agree. I generally believe that documentation should be kept in sync
> with releases (and I also think that the effort of doing so is minimal).
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101007/b89e51d8/attachment.html>

From skenny at uchicago.edu  Thu Oct  7 14:42:52 2010
From: skenny at uchicago.edu (Sarah Kenny)
Date: Thu, 7 Oct 2010 14:42:52 -0500
Subject: [Swift-devel] more on swift doc
Message-ID: <AANLkTik99JuwWU2aD+iqsk6erP6+T6j04tEhtLLYh4_=@mail.gmail.com>

does anyone have an objection to removing the 'really quick start guide'
from the user guide?
http://www.ci.uchicago.edu/swift/guides/reallyquickstartguide.php

i ask bcs a) we have a quick start guide which seems sufficiently succinct
to me and b) the files referenced in the 'really quick start guide', from
what i can tell, do not exist in our current,stable swift

?

~sk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101007/a4653313/attachment.html>

From hategan at mcs.anl.gov  Thu Oct  7 14:42:14 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 07 Oct 2010 12:42:14 -0700
Subject: [Swift-devel] swift versions
In-Reply-To: <AANLkTi=Z3QH0uSJrVN0d-FuBh5AnDCX_uoOqvbWkkp86@mail.gmail.com>
References: <AANLkTinGjCYiwN6bRnWXO5mOMPB0B9MEgQJL7mBwfVTB@mail.gmail.com>
	<1286331753.19448.6.camel@blabla2.none>
	<alpine.WNT.2.00.1010071102160.3252@justinwozniak>
	<AANLkTimMQuWc7DUSLf=EgsZWU=FvLsC-nhd154z4PvyF@mail.gmail.com>
	<alpine.DEB.2.00.1010071250240.1793@wozniak-desktop.mcs.anl.gov>
	<1286474908.25937.1.camel@blabla2.none>
	<AANLkTimGXuDPPbXoDDupsnWMaTREdBEuJ9rugVWcRrDQ@mail.gmail.com>
	<alpine.DEB.2.00.1010071419540.1793@wozniak-desktop.mcs.anl.gov>
	<1286480040.26601.0.camel@blabla2.none>
	<AANLkTi=Z3QH0uSJrVN0d-FuBh5AnDCX_uoOqvbWkkp86@mail.gmail.com>
Message-ID: <1286480534.26889.0.camel@blabla2.none>

On Thu, 2010-10-07 at 14:39 -0500, Sarah Kenny wrote:
> so, in this scenario, the changes to the doc that exist in each branch
> are pushed to the main user doc when we do the release or am i missing
> a step here?

That or we really have no "main doc" and instead we link from every
release. Though I feel odd about that.
> 
> On Thu, Oct 7, 2010 at 2:34 PM, Mihael Hategan <hategan at mcs.anl.gov>
> wrote:
>         
>         On Thu, 2010-10-07 at 14:28 -0500, Justin M Wozniak wrote:
>         > On Thu, 7 Oct 2010, Sarah Kenny wrote:
>         >
>         > > On Thu, Oct 7, 2010 at 1:08 PM, Mihael Hategan
>         <hategan at mcs.anl.gov> wrote:
>         > >
>         > >> Right. I think 1.0/4.1.7 should go out soon.
>         > >
>         > > ok, so i guess we should decide what 'soon' means ;) i am
>         currently going
>         > > thru the old bugs in bugzilla (at least trying to close
>         out things that have
>         > > been already fixed or are no-longer applicable, etc), but
>         perhaps it would
>         > > be good to determine if there are bigger issues outside of
>         that that still
>         > > need to be dealt with before we can put what we've got
>         into a stable release
>         > > and determine a time-frame...anything come to mind?
>         > >
>         > > as far as documentation...does it make sense for each
>         branch to have a full
>         > > copy of /ci/www/projects/swift under it which can then be
>         merged with the
>         > > main/live copy whenever the code is merged? admittedly, i
>         know nothing about
>         > > docbook, but from the standpoint of updating and merging
>         this seems to make
>         > > sense to me (though feel free to suggest another way :)
>         > >
>         > > ~sk
>         >
>         > I was looking at the update.sh script earlier today- I
>         propose we have one
>         > web site but multiple docs/guides directories, all
>         accessible from the
>         > docs/index.php page.  Each of these would be associated with
>         a branch.
>         > So, similar to the existing "Historical" section but for
>         "future" branches
>         > as well.  That would take a small modification to the
>         update.sh script and
>         > manual modification of the docs/index.php page for each
>         version number.
>         >
>         > We may also want to have the feature changes (past and
>         future version
>         > numbers) available on that page but I think those can be
>         plain text.
>         > These could be pulled directly from SVN as well.
>         >
>         
>         
>         I agree. I generally believe that documentation should be kept
>         in sync
>         with releases (and I also think that the effort of doing so is
>         minimal).
>         
> 




From skenny at uchicago.edu  Thu Oct  7 14:55:16 2010
From: skenny at uchicago.edu (Sarah Kenny)
Date: Thu, 7 Oct 2010 14:55:16 -0500
Subject: [Swift-devel] swift versions
In-Reply-To: <1286480534.26889.0.camel@blabla2.none>
References: <AANLkTinGjCYiwN6bRnWXO5mOMPB0B9MEgQJL7mBwfVTB@mail.gmail.com>
	<1286331753.19448.6.camel@blabla2.none>
	<alpine.WNT.2.00.1010071102160.3252@justinwozniak>
	<AANLkTimMQuWc7DUSLf=EgsZWU=FvLsC-nhd154z4PvyF@mail.gmail.com>
	<alpine.DEB.2.00.1010071250240.1793@wozniak-desktop.mcs.anl.gov>
	<1286474908.25937.1.camel@blabla2.none>
	<AANLkTimGXuDPPbXoDDupsnWMaTREdBEuJ9rugVWcRrDQ@mail.gmail.com>
	<alpine.DEB.2.00.1010071419540.1793@wozniak-desktop.mcs.anl.gov>
	<1286480040.26601.0.camel@blabla2.none>
	<AANLkTi=Z3QH0uSJrVN0d-FuBh5AnDCX_uoOqvbWkkp86@mail.gmail.com>
	<1286480534.26889.0.camel@blabla2.none>
Message-ID: <AANLkTinAavsNN0tXozgeE7JL849nGGk=WG9e+v0m=swJ@mail.gmail.com>

so, in that case the 'main user doc' would be something *like*
http://www.ci.uchicago.edu/swift/docs10/index.php ?

and THAT would include the updates from all the current branches merged into
it once we do a release?


On Thu, Oct 7, 2010 at 2:42 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> On Thu, 2010-10-07 at 14:39 -0500, Sarah Kenny wrote:
> > so, in this scenario, the changes to the doc that exist in each branch
> > are pushed to the main user doc when we do the release or am i missing
> > a step here?
>
> That or we really have no "main doc" and instead we link from every
> release. Though I feel odd about that.
> >
> > On Thu, Oct 7, 2010 at 2:34 PM, Mihael Hategan <hategan at mcs.anl.gov>
> > wrote:
> >
> >         On Thu, 2010-10-07 at 14:28 -0500, Justin M Wozniak wrote:
> >         > On Thu, 7 Oct 2010, Sarah Kenny wrote:
> >         >
> >         > > On Thu, Oct 7, 2010 at 1:08 PM, Mihael Hategan
> >         <hategan at mcs.anl.gov> wrote:
> >         > >
> >         > >> Right. I think 1.0/4.1.7 should go out soon.
> >         > >
> >         > > ok, so i guess we should decide what 'soon' means ;) i am
> >         currently going
> >         > > thru the old bugs in bugzilla (at least trying to close
> >         out things that have
> >         > > been already fixed or are no-longer applicable, etc), but
> >         perhaps it would
> >         > > be good to determine if there are bigger issues outside of
> >         that that still
> >         > > need to be dealt with before we can put what we've got
> >         into a stable release
> >         > > and determine a time-frame...anything come to mind?
> >         > >
> >         > > as far as documentation...does it make sense for each
> >         branch to have a full
> >         > > copy of /ci/www/projects/swift under it which can then be
> >         merged with the
> >         > > main/live copy whenever the code is merged? admittedly, i
> >         know nothing about
> >         > > docbook, but from the standpoint of updating and merging
> >         this seems to make
> >         > > sense to me (though feel free to suggest another way :)
> >         > >
> >         > > ~sk
> >         >
> >         > I was looking at the update.sh script earlier today- I
> >         propose we have one
> >         > web site but multiple docs/guides directories, all
> >         accessible from the
> >         > docs/index.php page.  Each of these would be associated with
> >         a branch.
> >         > So, similar to the existing "Historical" section but for
> >         "future" branches
> >         > as well.  That would take a small modification to the
> >         update.sh script and
> >         > manual modification of the docs/index.php page for each
> >         version number.
> >         >
> >         > We may also want to have the feature changes (past and
> >         future version
> >         > numbers) available on that page but I think those can be
> >         plain text.
> >         > These could be pulled directly from SVN as well.
> >         >
> >
> >
> >         I agree. I generally believe that documentation should be kept
> >         in sync
> >         with releases (and I also think that the effort of doing so is
> >         minimal).
> >
> >
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101007/054b371e/attachment.html>

From bugzilla-daemon at mcs.anl.gov  Thu Oct  7 15:14:02 2010
From: bugzilla-daemon at mcs.anl.gov (bugzilla-daemon at mcs.anl.gov)
Date: Thu,  7 Oct 2010 15:14:02 -0500 (CDT)
Subject: [Swift-devel] [Bug 106] Improve error messages for double-set and
	un-set variables
In-Reply-To: <bug-106-21@http.bugzilla.mcs.anl.gov/swift/>
References: <bug-106-21@http.bugzilla.mcs.anl.gov/swift/>
Message-ID: <20101007201402.918262C9F9@wind.mcs.anl.gov>

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=106


skenny <skenny at uchicago.edu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
                 CC|                            |skenny at uchicago.edu
         Resolution|                            |WORKSFORME




-- 
Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching the reporter.


From hategan at mcs.anl.gov  Thu Oct  7 15:17:59 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 07 Oct 2010 13:17:59 -0700
Subject: [Swift-devel] swift versions
In-Reply-To: <AANLkTinAavsNN0tXozgeE7JL849nGGk=WG9e+v0m=swJ@mail.gmail.com>
References: <AANLkTinGjCYiwN6bRnWXO5mOMPB0B9MEgQJL7mBwfVTB@mail.gmail.com>
	<1286331753.19448.6.camel@blabla2.none>
	<alpine.WNT.2.00.1010071102160.3252@justinwozniak>
	<AANLkTimMQuWc7DUSLf=EgsZWU=FvLsC-nhd154z4PvyF@mail.gmail.com>
	<alpine.DEB.2.00.1010071250240.1793@wozniak-desktop.mcs.anl.gov>
	<1286474908.25937.1.camel@blabla2.none>
	<AANLkTimGXuDPPbXoDDupsnWMaTREdBEuJ9rugVWcRrDQ@mail.gmail.com>
	<alpine.DEB.2.00.1010071419540.1793@wozniak-desktop.mcs.anl.gov>
	<1286480040.26601.0.camel@blabla2.none>
	<AANLkTi=Z3QH0uSJrVN0d-FuBh5AnDCX_uoOqvbWkkp86@mail.gmail.com>
	<1286480534.26889.0.camel@blabla2.none>
	<AANLkTinAavsNN0tXozgeE7JL849nGGk=WG9e+v0m=swJ@mail.gmail.com>
Message-ID: <1286482679.27212.0.camel@blabla2.none>

On Thu, 2010-10-07 at 14:55 -0500, Sarah Kenny wrote:
> so, in that case the 'main user doc' would be something *like*
> http://www.ci.uchicago.edu/swift/docs10/index.php ?
> 
> and THAT would include the updates from all the current branches

define "all current branches". We have:
1. Release branches
2. Trunk
3. Development branches (which are transient entities and only there to
make trunk's life easier).

>  merged into it once we do a release?
> 
> 
> On Thu, Oct 7, 2010 at 2:42 PM, Mihael Hategan <hategan at mcs.anl.gov>
> wrote:
>         On Thu, 2010-10-07 at 14:39 -0500, Sarah Kenny wrote:
>         > so, in this scenario, the changes to the doc that exist in
>         each branch
>         > are pushed to the main user doc when we do the release or am
>         i missing
>         > a step here?
>         
>         
>         That or we really have no "main doc" and instead we link from
>         every
>         release. Though I feel odd about that.
>         
>         >
>         > On Thu, Oct 7, 2010 at 2:34 PM, Mihael Hategan
>         <hategan at mcs.anl.gov>
>         > wrote:
>         >
>         >         On Thu, 2010-10-07 at 14:28 -0500, Justin M Wozniak
>         wrote:
>         >         > On Thu, 7 Oct 2010, Sarah Kenny wrote:
>         >         >
>         >         > > On Thu, Oct 7, 2010 at 1:08 PM, Mihael Hategan
>         >         <hategan at mcs.anl.gov> wrote:
>         >         > >
>         >         > >> Right. I think 1.0/4.1.7 should go out soon.
>         >         > >
>         >         > > ok, so i guess we should decide what 'soon'
>         means ;) i am
>         >         currently going
>         >         > > thru the old bugs in bugzilla (at least trying
>         to close
>         >         out things that have
>         >         > > been already fixed or are no-longer applicable,
>         etc), but
>         >         perhaps it would
>         >         > > be good to determine if there are bigger issues
>         outside of
>         >         that that still
>         >         > > need to be dealt with before we can put what
>         we've got
>         >         into a stable release
>         >         > > and determine a time-frame...anything come to
>         mind?
>         >         > >
>         >         > > as far as documentation...does it make sense for
>         each
>         >         branch to have a full
>         >         > > copy of /ci/www/projects/swift under it which
>         can then be
>         >         merged with the
>         >         > > main/live copy whenever the code is merged?
>         admittedly, i
>         >         know nothing about
>         >         > > docbook, but from the standpoint of updating and
>         merging
>         >         this seems to make
>         >         > > sense to me (though feel free to suggest another
>         way :)
>         >         > >
>         >         > > ~sk
>         >         >
>         >         > I was looking at the update.sh script earlier
>         today- I
>         >         propose we have one
>         >         > web site but multiple docs/guides directories, all
>         >         accessible from the
>         >         > docs/index.php page.  Each of these would be
>         associated with
>         >         a branch.
>         >         > So, similar to the existing "Historical" section
>         but for
>         >         "future" branches
>         >         > as well.  That would take a small modification to
>         the
>         >         update.sh script and
>         >         > manual modification of the docs/index.php page for
>         each
>         >         version number.
>         >         >
>         >         > We may also want to have the feature changes (past
>         and
>         >         future version
>         >         > numbers) available on that page but I think those
>         can be
>         >         plain text.
>         >         > These could be pulled directly from SVN as well.
>         >         >
>         >
>         >
>         >         I agree. I generally believe that documentation
>         should be kept
>         >         in sync
>         >         with releases (and I also think that the effort of
>         doing so is
>         >         minimal).
>         >
>         >
>         
>         
>         
> 




From skenny at uchicago.edu  Thu Oct  7 15:24:34 2010
From: skenny at uchicago.edu (Sarah Kenny)
Date: Thu, 7 Oct 2010 15:24:34 -0500
Subject: [Swift-devel] swift versions
In-Reply-To: <1286482679.27212.0.camel@blabla2.none>
References: <AANLkTinGjCYiwN6bRnWXO5mOMPB0B9MEgQJL7mBwfVTB@mail.gmail.com>
	<1286331753.19448.6.camel@blabla2.none>
	<alpine.WNT.2.00.1010071102160.3252@justinwozniak>
	<AANLkTimMQuWc7DUSLf=EgsZWU=FvLsC-nhd154z4PvyF@mail.gmail.com>
	<alpine.DEB.2.00.1010071250240.1793@wozniak-desktop.mcs.anl.gov>
	<1286474908.25937.1.camel@blabla2.none>
	<AANLkTimGXuDPPbXoDDupsnWMaTREdBEuJ9rugVWcRrDQ@mail.gmail.com>
	<alpine.DEB.2.00.1010071419540.1793@wozniak-desktop.mcs.anl.gov>
	<1286480040.26601.0.camel@blabla2.none>
	<AANLkTi=Z3QH0uSJrVN0d-FuBh5AnDCX_uoOqvbWkkp86@mail.gmail.com>
	<1286480534.26889.0.camel@blabla2.none>
	<AANLkTinAavsNN0tXozgeE7JL849nGGk=WG9e+v0m=swJ@mail.gmail.com>
	<1286482679.27212.0.camel@blabla2.none>
Message-ID: <AANLkTi=+djZuouVyS2AQDkz++iyZe6GhXvWAnmVfLtMR@mail.gmail.com>

 was thinking of what justin said, "I propose we have one web site but
multiple docs/guides directories, all accessible from the docs/index.php
page.  Each of these would be associated with a branch"

i was assuming that whatever branch(es) these were associated with, that doc
would somehow need to make its way to a main guide that we are pointing
users to.

?


On Thu, Oct 7, 2010 at 3:17 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> On Thu, 2010-10-07 at 14:55 -0500, Sarah Kenny wrote:
> > so, in that case the 'main user doc' would be something *like*
> > http://www.ci.uchicago.edu/swift/docs10/index.php ?
> >
> > and THAT would include the updates from all the current branches
>
> define "all current branches". We have:
> 1. Release branches
> 2. Trunk
> 3. Development branches (which are transient entities and only there to
> make trunk's life easier).
>
> >  merged into it once we do a release?
> >
> >
> > On Thu, Oct 7, 2010 at 2:42 PM, Mihael Hategan <hategan at mcs.anl.gov>
> > wrote:
> >         On Thu, 2010-10-07 at 14:39 -0500, Sarah Kenny wrote:
> >         > so, in this scenario, the changes to the doc that exist in
> >         each branch
> >         > are pushed to the main user doc when we do the release or am
> >         i missing
> >         > a step here?
> >
> >
> >         That or we really have no "main doc" and instead we link from
> >         every
> >         release. Though I feel odd about that.
> >
> >         >
> >         > On Thu, Oct 7, 2010 at 2:34 PM, Mihael Hategan
> >         <hategan at mcs.anl.gov>
> >         > wrote:
> >         >
> >         >         On Thu, 2010-10-07 at 14:28 -0500, Justin M Wozniak
> >         wrote:
> >         >         > On Thu, 7 Oct 2010, Sarah Kenny wrote:
> >         >         >
> >         >         > > On Thu, Oct 7, 2010 at 1:08 PM, Mihael Hategan
> >         >         <hategan at mcs.anl.gov> wrote:
> >         >         > >
> >         >         > >> Right. I think 1.0/4.1.7 should go out soon.
> >         >         > >
> >         >         > > ok, so i guess we should decide what 'soon'
> >         means ;) i am
> >         >         currently going
> >         >         > > thru the old bugs in bugzilla (at least trying
> >         to close
> >         >         out things that have
> >         >         > > been already fixed or are no-longer applicable,
> >         etc), but
> >         >         perhaps it would
> >         >         > > be good to determine if there are bigger issues
> >         outside of
> >         >         that that still
> >         >         > > need to be dealt with before we can put what
> >         we've got
> >         >         into a stable release
> >         >         > > and determine a time-frame...anything come to
> >         mind?
> >         >         > >
> >         >         > > as far as documentation...does it make sense for
> >         each
> >         >         branch to have a full
> >         >         > > copy of /ci/www/projects/swift under it which
> >         can then be
> >         >         merged with the
> >         >         > > main/live copy whenever the code is merged?
> >         admittedly, i
> >         >         know nothing about
> >         >         > > docbook, but from the standpoint of updating and
> >         merging
> >         >         this seems to make
> >         >         > > sense to me (though feel free to suggest another
> >         way :)
> >         >         > >
> >         >         > > ~sk
> >         >         >
> >         >         > I was looking at the update.sh script earlier
> >         today- I
> >         >         propose we have one
> >         >         > web site but multiple docs/guides directories, all
> >         >         accessible from the
> >         >         > docs/index.php page.  Each of these would be
> >         associated with
> >         >         a branch.
> >         >         > So, similar to the existing "Historical" section
> >         but for
> >         >         "future" branches
> >         >         > as well.  That would take a small modification to
> >         the
> >         >         update.sh script and
> >         >         > manual modification of the docs/index.php page for
> >         each
> >         >         version number.
> >         >         >
> >         >         > We may also want to have the feature changes (past
> >         and
> >         >         future version
> >         >         > numbers) available on that page but I think those
> >         can be
> >         >         plain text.
> >         >         > These could be pulled directly from SVN as well.
> >         >         >
> >         >
> >         >
> >         >         I agree. I generally believe that documentation
> >         should be kept
> >         >         in sync
> >         >         with releases (and I also think that the effort of
> >         doing so is
> >         >         minimal).
> >         >
> >         >
> >
> >
> >
> >
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101007/f931bfc9/attachment.html>

From wozniak at mcs.anl.gov  Thu Oct  7 15:30:49 2010
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Thu, 7 Oct 2010 15:30:49 -0500 (CDT)
Subject: [Swift-devel] swift versions
In-Reply-To: <AANLkTi=+djZuouVyS2AQDkz++iyZe6GhXvWAnmVfLtMR@mail.gmail.com>
References: <AANLkTinGjCYiwN6bRnWXO5mOMPB0B9MEgQJL7mBwfVTB@mail.gmail.com>
	<1286331753.19448.6.camel@blabla2.none>
	<alpine.WNT.2.00.1010071102160.3252@justinwozniak>
	<AANLkTimMQuWc7DUSLf=EgsZWU=FvLsC-nhd154z4PvyF@mail.gmail.com>
	<alpine.DEB.2.00.1010071250240.1793@wozniak-desktop.mcs.anl.gov>
	<1286474908.25937.1.camel@blabla2.none>
	<AANLkTimGXuDPPbXoDDupsnWMaTREdBEuJ9rugVWcRrDQ@mail.gmail.com>
	<alpine.DEB.2.00.1010071419540.1793@wozniak-desktop.mcs.anl.gov>
	<1286480040.26601.0.camel@blabla2.none>
	<AANLkTi=Z3QH0uSJrVN0d-FuBh5AnDCX_uoOqvbWkkp86@mail.gmail.com>
	<1286480534.26889.0.camel@blabla2.none>
	<AANLkTinAavsNN0tXozgeE7JL849nGGk=WG9e+v0m=swJ@mail.gmail.com>
	<1286482679.27212.0.camel@blabla2.none>
	<AANLkTi=+djZuouVyS2AQDkz++iyZe6GhXvWAnmVfLtMR@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1010071528230.1793@wozniak-desktop.mcs.anl.gov>


I meant release branch.  The valid branches could be hard-coded into the 
update.sh script.  The main guide would be the doc associated with the 
current version.  So right now, "main guide" would be aliased to 0.9 .

On Thu, 7 Oct 2010, Sarah Kenny wrote:

> was thinking of what justin said, "I propose we have one web site but
> multiple docs/guides directories, all accessible from the docs/index.php
> page.  Each of these would be associated with a branch"
>
> i was assuming that whatever branch(es) these were associated with, that doc
> would somehow need to make its way to a main guide that we are pointing
> users to.
>
> ?
>
>
> On Thu, Oct 7, 2010 at 3:17 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>
>> On Thu, 2010-10-07 at 14:55 -0500, Sarah Kenny wrote:
>>> so, in that case the 'main user doc' would be something *like*
>>> http://www.ci.uchicago.edu/swift/docs10/index.php ?
>>>
>>> and THAT would include the updates from all the current branches
>>
>> define "all current branches". We have:
>> 1. Release branches
>> 2. Trunk
>> 3. Development branches (which are transient entities and only there to
>> make trunk's life easier).
>>
>>>  merged into it once we do a release?
>>>
>>>
>>> On Thu, Oct 7, 2010 at 2:42 PM, Mihael Hategan <hategan at mcs.anl.gov>
>>> wrote:
>>>         On Thu, 2010-10-07 at 14:39 -0500, Sarah Kenny wrote:
>>>        > so, in this scenario, the changes to the doc that exist in
>>>         each branch
>>>        > are pushed to the main user doc when we do the release or am
>>>         i missing
>>>        > a step here?
>>>
>>>
>>>         That or we really have no "main doc" and instead we link from
>>>         every
>>>         release. Though I feel odd about that.
>>>
>>>        >
>>>        > On Thu, Oct 7, 2010 at 2:34 PM, Mihael Hategan
>>>         <hategan at mcs.anl.gov>
>>>        > wrote:
>>>        >
>>>        >         On Thu, 2010-10-07 at 14:28 -0500, Justin M Wozniak
>>>         wrote:
>>>        >        > On Thu, 7 Oct 2010, Sarah Kenny wrote:
>>>        >        >
>>>        >        >> On Thu, Oct 7, 2010 at 1:08 PM, Mihael Hategan
>>>        >         <hategan at mcs.anl.gov> wrote:
>>>        >        >>
>>>        >        >>> Right. I think 1.0/4.1.7 should go out soon.
>>>        >        >>
>>>        >        >> ok, so i guess we should decide what 'soon'
>>>         means ;) i am
>>>        >         currently going
>>>        >        >> thru the old bugs in bugzilla (at least trying
>>>         to close
>>>        >         out things that have
>>>        >        >> been already fixed or are no-longer applicable,
>>>         etc), but
>>>        >         perhaps it would
>>>        >        >> be good to determine if there are bigger issues
>>>         outside of
>>>        >         that that still
>>>        >        >> need to be dealt with before we can put what
>>>         we've got
>>>        >         into a stable release
>>>        >        >> and determine a time-frame...anything come to
>>>         mind?
>>>        >        >>
>>>        >        >> as far as documentation...does it make sense for
>>>         each
>>>        >         branch to have a full
>>>        >        >> copy of /ci/www/projects/swift under it which
>>>         can then be
>>>        >         merged with the
>>>        >        >> main/live copy whenever the code is merged?
>>>         admittedly, i
>>>        >         know nothing about
>>>        >        >> docbook, but from the standpoint of updating and
>>>         merging
>>>        >         this seems to make
>>>        >        >> sense to me (though feel free to suggest another
>>>         way :)
>>>        >        >>
>>>        >        >> ~sk
>>>        >        >
>>>        >        > I was looking at the update.sh script earlier
>>>         today- I
>>>        >         propose we have one
>>>        >        > web site but multiple docs/guides directories, all
>>>        >         accessible from the
>>>        >        > docs/index.php page.  Each of these would be
>>>         associated with
>>>        >         a branch.
>>>        >        > So, similar to the existing "Historical" section
>>>         but for
>>>        >         "future" branches
>>>        >        > as well.  That would take a small modification to
>>>         the
>>>        >         update.sh script and
>>>        >        > manual modification of the docs/index.php page for
>>>         each
>>>        >         version number.
>>>        >        >
>>>        >        > We may also want to have the feature changes (past
>>>         and
>>>        >         future version
>>>        >        > numbers) available on that page but I think those
>>>         can be
>>>        >         plain text.
>>>        >        > These could be pulled directly from SVN as well.
>>>        >        >
>>>        >
>>>        >
>>>        >         I agree. I generally believe that documentation
>>>         should be kept
>>>        >         in sync
>>>        >         with releases (and I also think that the effort of
>>>         doing so is
>>>        >         minimal).
>>>        >
>>>        >
>>>
>>>
>>>
>>>
>>
>>
>>
>

-- 
Justin M Wozniak


From bugzilla-daemon at mcs.anl.gov  Thu Oct  7 15:36:37 2010
From: bugzilla-daemon at mcs.anl.gov (bugzilla-daemon at mcs.anl.gov)
Date: Thu,  7 Oct 2010 15:36:37 -0500 (CDT)
Subject: [Swift-devel] [Bug 28] documentation section #links should be
 semantically meaningful
In-Reply-To: <bug-28-21@http.bugzilla.mcs.anl.gov/swift/>
References: <bug-28-21@http.bugzilla.mcs.anl.gov/swift/>
Message-ID: <20101007203637.68BAC2C9FB@wind.mcs.anl.gov>

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=28


skenny <skenny at uchicago.edu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |skenny at uchicago.edu
         Resolution|                            |WONTFIX




-- 
Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching the reporter.


From hategan at mcs.anl.gov  Thu Oct  7 15:39:56 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 07 Oct 2010 13:39:56 -0700
Subject: [Swift-devel] swift versions
In-Reply-To: <alpine.DEB.2.00.1010071528230.1793@wozniak-desktop.mcs.anl.gov>
References: <AANLkTinGjCYiwN6bRnWXO5mOMPB0B9MEgQJL7mBwfVTB@mail.gmail.com>
	<1286331753.19448.6.camel@blabla2.none>
	<alpine.WNT.2.00.1010071102160.3252@justinwozniak>
	<AANLkTimMQuWc7DUSLf=EgsZWU=FvLsC-nhd154z4PvyF@mail.gmail.com>
	<alpine.DEB.2.00.1010071250240.1793@wozniak-desktop.mcs.anl.gov>
	<1286474908.25937.1.camel@blabla2.none>
	<AANLkTimGXuDPPbXoDDupsnWMaTREdBEuJ9rugVWcRrDQ@mail.gmail.com>
	<alpine.DEB.2.00.1010071419540.1793@wozniak-desktop.mcs.anl.gov>
	<1286480040.26601.0.camel@blabla2.none>
	<AANLkTi=Z3QH0uSJrVN0d-FuBh5AnDCX_uoOqvbWkkp86@mail.gmail.com>
	<1286480534.26889.0.camel@blabla2.none>
	<AANLkTinAavsNN0tXozgeE7JL849nGGk=WG9e+v0m=swJ@mail.gmail.com>
	<1286482679.27212.0.camel@blabla2.none>
	<AANLkTi=+djZuouVyS2AQDkz++iyZe6GhXvWAnmVfLtMR@mail.gmail.com>
	<alpine.DEB.2.00.1010071528230.1793@wozniak-desktop.mcs.anl.gov>
Message-ID: <1286483996.27434.2.camel@blabla2.none>

To summarize:

There is a 1-1 mapping between branches and docs. One of the branches
(corresponding to the current release) gets linked from "main" (i.e.
main docs are the docs for the current release). So:

branches/0.9 <-> 0.9 docs
banches/0.8 <-> 0.8 docs
trunk <-> trunk docs

If current release is 0.9, then main docs = 0.9 docs.

On Thu, 2010-10-07 at 15:30 -0500, Justin M Wozniak wrote:
> I meant release branch.  The valid branches could be hard-coded into the 
> update.sh script.  The main guide would be the doc associated with the 
> current version.  So right now, "main guide" would be aliased to 0.9 .
> 
> On Thu, 7 Oct 2010, Sarah Kenny wrote:
> 
> > was thinking of what justin said, "I propose we have one web site but
> > multiple docs/guides directories, all accessible from the docs/index.php
> > page.  Each of these would be associated with a branch"
> >
> > i was assuming that whatever branch(es) these were associated with, that doc
> > would somehow need to make its way to a main guide that we are pointing
> > users to.
> >
> > ?
> >
> >
> > On Thu, Oct 7, 2010 at 3:17 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> >
> >> On Thu, 2010-10-07 at 14:55 -0500, Sarah Kenny wrote:
> >>> so, in that case the 'main user doc' would be something *like*
> >>> http://www.ci.uchicago.edu/swift/docs10/index.php ?
> >>>
> >>> and THAT would include the updates from all the current branches
> >>
> >> define "all current branches". We have:
> >> 1. Release branches
> >> 2. Trunk
> >> 3. Development branches (which are transient entities and only there to
> >> make trunk's life easier).
> >>
> >>>  merged into it once we do a release?
> >>>
> >>>
> >>> On Thu, Oct 7, 2010 at 2:42 PM, Mihael Hategan <hategan at mcs.anl.gov>
> >>> wrote:
> >>>         On Thu, 2010-10-07 at 14:39 -0500, Sarah Kenny wrote:
> >>>        > so, in this scenario, the changes to the doc that exist in
> >>>         each branch
> >>>        > are pushed to the main user doc when we do the release or am
> >>>         i missing
> >>>        > a step here?
> >>>
> >>>
> >>>         That or we really have no "main doc" and instead we link from
> >>>         every
> >>>         release. Though I feel odd about that.
> >>>
> >>>        >
> >>>        > On Thu, Oct 7, 2010 at 2:34 PM, Mihael Hategan
> >>>         <hategan at mcs.anl.gov>
> >>>        > wrote:
> >>>        >
> >>>        >         On Thu, 2010-10-07 at 14:28 -0500, Justin M Wozniak
> >>>         wrote:
> >>>        >        > On Thu, 7 Oct 2010, Sarah Kenny wrote:
> >>>        >        >
> >>>        >        >> On Thu, Oct 7, 2010 at 1:08 PM, Mihael Hategan
> >>>        >         <hategan at mcs.anl.gov> wrote:
> >>>        >        >>
> >>>        >        >>> Right. I think 1.0/4.1.7 should go out soon.
> >>>        >        >>
> >>>        >        >> ok, so i guess we should decide what 'soon'
> >>>         means ;) i am
> >>>        >         currently going
> >>>        >        >> thru the old bugs in bugzilla (at least trying
> >>>         to close
> >>>        >         out things that have
> >>>        >        >> been already fixed or are no-longer applicable,
> >>>         etc), but
> >>>        >         perhaps it would
> >>>        >        >> be good to determine if there are bigger issues
> >>>         outside of
> >>>        >         that that still
> >>>        >        >> need to be dealt with before we can put what
> >>>         we've got
> >>>        >         into a stable release
> >>>        >        >> and determine a time-frame...anything come to
> >>>         mind?
> >>>        >        >>
> >>>        >        >> as far as documentation...does it make sense for
> >>>         each
> >>>        >         branch to have a full
> >>>        >        >> copy of /ci/www/projects/swift under it which
> >>>         can then be
> >>>        >         merged with the
> >>>        >        >> main/live copy whenever the code is merged?
> >>>         admittedly, i
> >>>        >         know nothing about
> >>>        >        >> docbook, but from the standpoint of updating and
> >>>         merging
> >>>        >         this seems to make
> >>>        >        >> sense to me (though feel free to suggest another
> >>>         way :)
> >>>        >        >>
> >>>        >        >> ~sk
> >>>        >        >
> >>>        >        > I was looking at the update.sh script earlier
> >>>         today- I
> >>>        >         propose we have one
> >>>        >        > web site but multiple docs/guides directories, all
> >>>        >         accessible from the
> >>>        >        > docs/index.php page.  Each of these would be
> >>>         associated with
> >>>        >         a branch.
> >>>        >        > So, similar to the existing "Historical" section
> >>>         but for
> >>>        >         "future" branches
> >>>        >        > as well.  That would take a small modification to
> >>>         the
> >>>        >         update.sh script and
> >>>        >        > manual modification of the docs/index.php page for
> >>>         each
> >>>        >         version number.
> >>>        >        >
> >>>        >        > We may also want to have the feature changes (past
> >>>         and
> >>>        >         future version
> >>>        >        > numbers) available on that page but I think those
> >>>         can be
> >>>        >         plain text.
> >>>        >        > These could be pulled directly from SVN as well.
> >>>        >        >
> >>>        >
> >>>        >
> >>>        >         I agree. I generally believe that documentation
> >>>         should be kept
> >>>        >         in sync
> >>>        >         with releases (and I also think that the effort of
> >>>         doing so is
> >>>        >         minimal).
> >>>        >
> >>>        >
> >>>
> >>>
> >>>
> >>>
> >>
> >>
> >>
> >
> 




From wozniak at mcs.anl.gov  Thu Oct  7 15:45:54 2010
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Thu, 7 Oct 2010 15:45:54 -0500 (CDT)
Subject: [Swift-devel] swift versions
In-Reply-To: <1286483996.27434.2.camel@blabla2.none>
References: <AANLkTinGjCYiwN6bRnWXO5mOMPB0B9MEgQJL7mBwfVTB@mail.gmail.com>
	<1286331753.19448.6.camel@blabla2.none>
	<alpine.WNT.2.00.1010071102160.3252@justinwozniak>
	<AANLkTimMQuWc7DUSLf=EgsZWU=FvLsC-nhd154z4PvyF@mail.gmail.com> 
	<alpine.DEB.2.00.1010071250240.1793@wozniak-desktop.mcs.anl.gov>
	<1286474908.25937.1.camel@blabla2.none>
	<AANLkTimGXuDPPbXoDDupsnWMaTREdBEuJ9rugVWcRrDQ@mail.gmail.com>
	<alpine.DEB.2.00.1010071419540.1793@wozniak-desktop.mcs.anl.gov>
	<1286480040.26601.0.camel@blabla2.none>
	<AANLkTi=Z3QH0uSJrVN0d-FuBh5AnDCX_uoOqvbWkkp86@mail.gmail.com>
	<1286480534.26889.0.camel@blabla2.none>
	<AANLkTinAavsNN0tXozgeE7JL849nGGk=WG9e+v0m=swJ@mail.gmail.com>
	<1286482679.27212.0.camel@blabla2.none> 
	<AANLkTi=+djZuouVyS2AQDkz++iyZe6GhXvWAnmVfLtMR@mail.gmail.com>
	<alpine.DEB.2.00.1010071528230.1793@wozniak-desktop.mcs.anl.gov>
	<1286483996.27434.2.camel@blabla2.none>
Message-ID: <alpine.DEB.2.00.1010071545460.1793@wozniak-desktop.mcs.anl.gov>


Yup.

On Thu, 7 Oct 2010, Mihael Hategan wrote:

> To summarize:
>
> There is a 1-1 mapping between branches and docs. One of the branches
> (corresponding to the current release) gets linked from "main" (i.e.
> main docs are the docs for the current release). So:
>
> branches/0.9 <-> 0.9 docs
> banches/0.8 <-> 0.8 docs
> trunk <-> trunk docs
>
> If current release is 0.9, then main docs = 0.9 docs.
>
> On Thu, 2010-10-07 at 15:30 -0500, Justin M Wozniak wrote:
>> I meant release branch.  The valid branches could be hard-coded into the
>> update.sh script.  The main guide would be the doc associated with the
>> current version.  So right now, "main guide" would be aliased to 0.9 .
>>
>> On Thu, 7 Oct 2010, Sarah Kenny wrote:
>>
>>> was thinking of what justin said, "I propose we have one web site but
>>> multiple docs/guides directories, all accessible from the docs/index.php
>>> page.  Each of these would be associated with a branch"
>>>
>>> i was assuming that whatever branch(es) these were associated with, that doc
>>> would somehow need to make its way to a main guide that we are pointing
>>> users to.
>>>
>>> ?
>>>
>>>
>>> On Thu, Oct 7, 2010 at 3:17 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>>>
>>>> On Thu, 2010-10-07 at 14:55 -0500, Sarah Kenny wrote:
>>>>> so, in that case the 'main user doc' would be something *like*
>>>>> http://www.ci.uchicago.edu/swift/docs10/index.php ?
>>>>>
>>>>> and THAT would include the updates from all the current branches
>>>>
>>>> define "all current branches". We have:
>>>> 1. Release branches
>>>> 2. Trunk
>>>> 3. Development branches (which are transient entities and only there to
>>>> make trunk's life easier).
>>>>
>>>>>  merged into it once we do a release?
>>>>>
>>>>>
>>>>> On Thu, Oct 7, 2010 at 2:42 PM, Mihael Hategan <hategan at mcs.anl.gov>
>>>>> wrote:
>>>>>         On Thu, 2010-10-07 at 14:39 -0500, Sarah Kenny wrote:
>>>>>       > so, in this scenario, the changes to the doc that exist in
>>>>>         each branch
>>>>>       > are pushed to the main user doc when we do the release or am
>>>>>         i missing
>>>>>       > a step here?
>>>>>
>>>>>
>>>>>         That or we really have no "main doc" and instead we link from
>>>>>         every
>>>>>         release. Though I feel odd about that.
>>>>>
>>>>>       >
>>>>>       > On Thu, Oct 7, 2010 at 2:34 PM, Mihael Hategan
>>>>>         <hategan at mcs.anl.gov>
>>>>>       > wrote:
>>>>>       >
>>>>>       >         On Thu, 2010-10-07 at 14:28 -0500, Justin M Wozniak
>>>>>         wrote:
>>>>>       >       > On Thu, 7 Oct 2010, Sarah Kenny wrote:
>>>>>       >       >
>>>>>       >       >> On Thu, Oct 7, 2010 at 1:08 PM, Mihael Hategan
>>>>>       >         <hategan at mcs.anl.gov> wrote:
>>>>>       >       >>
>>>>>       >       >>> Right. I think 1.0/4.1.7 should go out soon.
>>>>>       >       >>
>>>>>       >       >> ok, so i guess we should decide what 'soon'
>>>>>         means ;) i am
>>>>>       >         currently going
>>>>>       >       >> thru the old bugs in bugzilla (at least trying
>>>>>         to close
>>>>>       >         out things that have
>>>>>       >       >> been already fixed or are no-longer applicable,
>>>>>         etc), but
>>>>>       >         perhaps it would
>>>>>       >       >> be good to determine if there are bigger issues
>>>>>         outside of
>>>>>       >         that that still
>>>>>       >       >> need to be dealt with before we can put what
>>>>>         we've got
>>>>>       >         into a stable release
>>>>>       >       >> and determine a time-frame...anything come to
>>>>>         mind?
>>>>>       >       >>
>>>>>       >       >> as far as documentation...does it make sense for
>>>>>         each
>>>>>       >         branch to have a full
>>>>>       >       >> copy of /ci/www/projects/swift under it which
>>>>>         can then be
>>>>>       >         merged with the
>>>>>       >       >> main/live copy whenever the code is merged?
>>>>>         admittedly, i
>>>>>       >         know nothing about
>>>>>       >       >> docbook, but from the standpoint of updating and
>>>>>         merging
>>>>>       >         this seems to make
>>>>>       >       >> sense to me (though feel free to suggest another
>>>>>         way :)
>>>>>       >       >>
>>>>>       >       >> ~sk
>>>>>       >       >
>>>>>       >       > I was looking at the update.sh script earlier
>>>>>         today- I
>>>>>       >         propose we have one
>>>>>       >       > web site but multiple docs/guides directories, all
>>>>>       >         accessible from the
>>>>>       >       > docs/index.php page.  Each of these would be
>>>>>         associated with
>>>>>       >         a branch.
>>>>>       >       > So, similar to the existing "Historical" section
>>>>>         but for
>>>>>       >         "future" branches
>>>>>       >       > as well.  That would take a small modification to
>>>>>         the
>>>>>       >         update.sh script and
>>>>>       >       > manual modification of the docs/index.php page for
>>>>>         each
>>>>>       >         version number.
>>>>>       >       >
>>>>>       >       > We may also want to have the feature changes (past
>>>>>         and
>>>>>       >         future version
>>>>>       >       > numbers) available on that page but I think those
>>>>>         can be
>>>>>       >         plain text.
>>>>>       >       > These could be pulled directly from SVN as well.
>>>>>       >       >
>>>>>       >
>>>>>       >
>>>>>       >         I agree. I generally believe that documentation
>>>>>         should be kept
>>>>>       >         in sync
>>>>>       >         with releases (and I also think that the effort of
>>>>>         doing so is
>>>>>       >         minimal).
>>>>>       >
>>>>>       >
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>
>
>
>

-- 
Justin M Wozniak


From wilde at mcs.anl.gov  Thu Oct  7 15:50:51 2010
From: wilde at mcs.anl.gov (wilde at mcs.anl.gov)
Date: Thu, 7 Oct 2010 14:50:51 -0600 (GMT-06:00)
Subject: [Swift-devel] more on swift doc
In-Reply-To: <747742188.849171286484408115.JavaMail.root@zimbra.anl.gov>
Message-ID: <995198347.849421286484651270.JavaMail.root@zimbra.anl.gov>

I have a slight preference for merging the "really quick start guide" in the Quick Start Guide. 

For now, perhaps just copy the source contents of Really Quick into Quick and comment it out; it would be nice to have a working version of the Really Quick steps in the future. 

- Mike 


----- "Sarah Kenny" <skenny at uchicago.edu> wrote: 
> does anyone have an objection to removing the 'really quick start guide' from the user guide? http://www.ci.uchicago.edu/swift/guides/reallyquickstartguide.php 
> 
> i ask bcs a) we have a quick start guide which seems sufficiently succinct to me and b) the files referenced in the 'really quick start guide', from what i can tell, do not exist in our current,stable swift 
> 
> ? 
> 
> ~sk 
> 
> _______________________________________________ Swift-devel mailing list Swift-devel at ci.uchicago.edu http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel 

-- 
Michael Wilde 
Computation Institute, University of Chicago 
Mathematics and Computer Science Division 
Argonne National Laboratory 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101007/ec1da992/attachment.html>

From skenny at uchicago.edu  Thu Oct  7 15:54:53 2010
From: skenny at uchicago.edu (Sarah Kenny)
Date: Thu, 7 Oct 2010 15:54:53 -0500
Subject: [Swift-devel] more on swift doc
In-Reply-To: <995198347.849421286484651270.JavaMail.root@zimbra.anl.gov>
References: <747742188.849171286484408115.JavaMail.root@zimbra.anl.gov>
	<995198347.849421286484651270.JavaMail.root@zimbra.anl.gov>
Message-ID: <AANLkTim06115GpRJKQ9asxAx-F=_q4WFwxhXrsgSeECQ@mail.gmail.com>

On Thu, Oct 7, 2010 at 3:50 PM, <wilde at mcs.anl.gov> wrote:

> I have a slight preference for merging the "really quick start guide" in
> the Quick Start Guide.
>
> For now, perhaps just copy the source contents of Really Quick into Quick
> and comment it out; it would be nice to have a working version of the Really
> Quick steps in the future.
>

will do...though i think this might be a form of code hording, haha



>
> - Mike
>
>
>
> ----- "Sarah Kenny" <skenny at uchicago.edu> wrote:
> > does anyone have an objection to removing the 'really quick start guide'
> from the user guide?
> http://www.ci.uchicago.edu/swift/guides/reallyquickstartguide.php
> >
> > i ask bcs a) we have a quick start guide which seems sufficiently
> succinct to me and b) the files referenced in the 'really quick start
> guide', from what i can tell, do not exist in our current,stable swift
> >
> > ?
> >
> > ~sk
> >
> > _______________________________________________ Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101007/0e32fea7/attachment.html>

From skenny at uchicago.edu  Thu Oct  7 16:00:19 2010
From: skenny at uchicago.edu (Sarah Kenny)
Date: Thu, 7 Oct 2010 16:00:19 -0500
Subject: [Swift-devel] swift versions
In-Reply-To: <alpine.DEB.2.00.1010071545460.1793@wozniak-desktop.mcs.anl.gov>
References: <AANLkTinGjCYiwN6bRnWXO5mOMPB0B9MEgQJL7mBwfVTB@mail.gmail.com>
	<1286331753.19448.6.camel@blabla2.none>
	<alpine.WNT.2.00.1010071102160.3252@justinwozniak>
	<AANLkTimMQuWc7DUSLf=EgsZWU=FvLsC-nhd154z4PvyF@mail.gmail.com>
	<alpine.DEB.2.00.1010071250240.1793@wozniak-desktop.mcs.anl.gov>
	<1286474908.25937.1.camel@blabla2.none>
	<AANLkTimGXuDPPbXoDDupsnWMaTREdBEuJ9rugVWcRrDQ@mail.gmail.com>
	<alpine.DEB.2.00.1010071419540.1793@wozniak-desktop.mcs.anl.gov>
	<1286480040.26601.0.camel@blabla2.none>
	<AANLkTi=Z3QH0uSJrVN0d-FuBh5AnDCX_uoOqvbWkkp86@mail.gmail.com>
	<1286480534.26889.0.camel@blabla2.none>
	<AANLkTinAavsNN0tXozgeE7JL849nGGk=WG9e+v0m=swJ@mail.gmail.com>
	<1286482679.27212.0.camel@blabla2.none>
	<AANLkTi=+djZuouVyS2AQDkz++iyZe6GhXvWAnmVfLtMR@mail.gmail.com>
	<alpine.DEB.2.00.1010071528230.1793@wozniak-desktop.mcs.anl.gov>
	<1286483996.27434.2.camel@blabla2.none>
	<alpine.DEB.2.00.1010071545460.1793@wozniak-desktop.mcs.anl.gov>
Message-ID: <AANLkTimp7MskwvD9aFy5qBKvR2--cJJmEKs68W6QFzLV@mail.gmail.com>

sounds good.

just want to verify (sorry to beat this to death):

so, we alias the 'main guide' to 0.9 (as suggested) and any changes we make
will go to the doc in branches/1.0 which will then be the new alias for the
'main guide' once we do the release.



On Thu, Oct 7, 2010 at 3:45 PM, Justin M Wozniak <wozniak at mcs.anl.gov>wrote:

>
> Yup.
>
>
> On Thu, 7 Oct 2010, Mihael Hategan wrote:
>
>  To summarize:
>>
>> There is a 1-1 mapping between branches and docs. One of the branches
>> (corresponding to the current release) gets linked from "main" (i.e.
>> main docs are the docs for the current release). So:
>>
>> branches/0.9 <-> 0.9 docs
>> banches/0.8 <-> 0.8 docs
>> trunk <-> trunk docs
>>
>> If current release is 0.9, then main docs = 0.9 docs.
>>
>> On Thu, 2010-10-07 at 15:30 -0500, Justin M Wozniak wrote:
>>
>>> I meant release branch.  The valid branches could be hard-coded into the
>>> update.sh script.  The main guide would be the doc associated with the
>>> current version.  So right now, "main guide" would be aliased to 0.9 .
>>>
>>> On Thu, 7 Oct 2010, Sarah Kenny wrote:
>>>
>>>  was thinking of what justin said, "I propose we have one web site but
>>>> multiple docs/guides directories, all accessible from the docs/index.php
>>>> page.  Each of these would be associated with a branch"
>>>>
>>>> i was assuming that whatever branch(es) these were associated with, that
>>>> doc
>>>> would somehow need to make its way to a main guide that we are pointing
>>>> users to.
>>>>
>>>> ?
>>>>
>>>>
>>>> On Thu, Oct 7, 2010 at 3:17 PM, Mihael Hategan <hategan at mcs.anl.gov>
>>>> wrote:
>>>>
>>>>  On Thu, 2010-10-07 at 14:55 -0500, Sarah Kenny wrote:
>>>>>
>>>>>> so, in that case the 'main user doc' would be something *like*
>>>>>> http://www.ci.uchicago.edu/swift/docs10/index.php ?
>>>>>>
>>>>>> and THAT would include the updates from all the current branches
>>>>>>
>>>>>
>>>>> define "all current branches". We have:
>>>>> 1. Release branches
>>>>> 2. Trunk
>>>>> 3. Development branches (which are transient entities and only there to
>>>>> make trunk's life easier).
>>>>>
>>>>>   merged into it once we do a release?
>>>>>>
>>>>>>
>>>>>> On Thu, Oct 7, 2010 at 2:42 PM, Mihael Hategan <hategan at mcs.anl.gov>
>>>>>> wrote:
>>>>>>        On Thu, 2010-10-07 at 14:39 -0500, Sarah Kenny wrote:
>>>>>>      > so, in this scenario, the changes to the doc that exist in
>>>>>>        each branch
>>>>>>      > are pushed to the main user doc when we do the release or am
>>>>>>        i missing
>>>>>>      > a step here?
>>>>>>
>>>>>>
>>>>>>        That or we really have no "main doc" and instead we link from
>>>>>>        every
>>>>>>        release. Though I feel odd about that.
>>>>>>
>>>>>>      >
>>>>>>      > On Thu, Oct 7, 2010 at 2:34 PM, Mihael Hategan
>>>>>>        <hategan at mcs.anl.gov>
>>>>>>      > wrote:
>>>>>>      >
>>>>>>      >         On Thu, 2010-10-07 at 14:28 -0500, Justin M Wozniak
>>>>>>        wrote:
>>>>>>      >       > On Thu, 7 Oct 2010, Sarah Kenny wrote:
>>>>>>      >       >
>>>>>>      >       >> On Thu, Oct 7, 2010 at 1:08 PM, Mihael Hategan
>>>>>>      >         <hategan at mcs.anl.gov> wrote:
>>>>>>      >       >>
>>>>>>      >       >>> Right. I think 1.0/4.1.7 should go out soon.
>>>>>>      >       >>
>>>>>>      >       >> ok, so i guess we should decide what 'soon'
>>>>>>        means ;) i am
>>>>>>      >         currently going
>>>>>>      >       >> thru the old bugs in bugzilla (at least trying
>>>>>>        to close
>>>>>>      >         out things that have
>>>>>>      >       >> been already fixed or are no-longer applicable,
>>>>>>        etc), but
>>>>>>      >         perhaps it would
>>>>>>      >       >> be good to determine if there are bigger issues
>>>>>>        outside of
>>>>>>      >         that that still
>>>>>>      >       >> need to be dealt with before we can put what
>>>>>>        we've got
>>>>>>      >         into a stable release
>>>>>>      >       >> and determine a time-frame...anything come to
>>>>>>        mind?
>>>>>>      >       >>
>>>>>>      >       >> as far as documentation...does it make sense for
>>>>>>        each
>>>>>>      >         branch to have a full
>>>>>>      >       >> copy of /ci/www/projects/swift under it which
>>>>>>        can then be
>>>>>>      >         merged with the
>>>>>>      >       >> main/live copy whenever the code is merged?
>>>>>>        admittedly, i
>>>>>>      >         know nothing about
>>>>>>      >       >> docbook, but from the standpoint of updating and
>>>>>>        merging
>>>>>>      >         this seems to make
>>>>>>      >       >> sense to me (though feel free to suggest another
>>>>>>        way :)
>>>>>>      >       >>
>>>>>>      >       >> ~sk
>>>>>>      >       >
>>>>>>      >       > I was looking at the update.sh script earlier
>>>>>>        today- I
>>>>>>      >         propose we have one
>>>>>>      >       > web site but multiple docs/guides directories, all
>>>>>>      >         accessible from the
>>>>>>      >       > docs/index.php page.  Each of these would be
>>>>>>        associated with
>>>>>>      >         a branch.
>>>>>>      >       > So, similar to the existing "Historical" section
>>>>>>        but for
>>>>>>      >         "future" branches
>>>>>>      >       > as well.  That would take a small modification to
>>>>>>        the
>>>>>>      >         update.sh script and
>>>>>>      >       > manual modification of the docs/index.php page for
>>>>>>        each
>>>>>>      >         version number.
>>>>>>      >       >
>>>>>>      >       > We may also want to have the feature changes (past
>>>>>>        and
>>>>>>      >         future version
>>>>>>      >       > numbers) available on that page but I think those
>>>>>>        can be
>>>>>>      >         plain text.
>>>>>>      >       > These could be pulled directly from SVN as well.
>>>>>>      >       >
>>>>>>      >
>>>>>>      >
>>>>>>      >         I agree. I generally believe that documentation
>>>>>>        should be kept
>>>>>>      >         in sync
>>>>>>      >         with releases (and I also think that the effort of
>>>>>>        doing so is
>>>>>>      >         minimal).
>>>>>>      >
>>>>>>      >
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>>
> --
> Justin M Wozniak
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101007/ce299a54/attachment.html>

From wozniak at mcs.anl.gov  Thu Oct  7 16:08:07 2010
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Thu, 7 Oct 2010 16:08:07 -0500 (CDT)
Subject: [Swift-devel] swift versions
In-Reply-To: <AANLkTimp7MskwvD9aFy5qBKvR2--cJJmEKs68W6QFzLV@mail.gmail.com>
References: <AANLkTinGjCYiwN6bRnWXO5mOMPB0B9MEgQJL7mBwfVTB@mail.gmail.com>
	<alpine.WNT.2.00.1010071102160.3252@justinwozniak>
	<AANLkTimMQuWc7DUSLf=EgsZWU=FvLsC-nhd154z4PvyF@mail.gmail.com>
	<alpine.DEB.2.00.1010071250240.1793@wozniak-desktop.mcs.anl.gov>
	<1286474908.25937.1.camel@blabla2.none>
	<AANLkTimGXuDPPbXoDDupsnWMaTREdBEuJ9rugVWcRrDQ@mail.gmail.com>
	<alpine.DEB.2.00.1010071419540.1793@wozniak-desktop.mcs.anl.gov>
	<1286480040.26601.0.camel@blabla2.none>
	<AANLkTi=Z3QH0uSJrVN0d-FuBh5AnDCX_uoOqvbWkkp86@mail.gmail.com>
	<1286480534.26889.0.camel@blabla2.none>
	<AANLkTinAavsNN0tXozgeE7JL849nGGk=WG9e+v0m=swJ@mail.gmail.com>
	<1286482679.27212.0.camel@blabla2.none>
	<AANLkTi=+djZuouVyS2AQDkz++iyZe6GhXvWAnmVfLtMR@mail.gmail.com>
	<alpine.DEB.2.00.1010071528230.1793@wozniak-desktop.mcs.anl.gov>
	<1286483996.27434.2.camel@blabla2.none>
	<alpine.DEB.2.00.1010071545460.1793@wozniak-desktop.mcs.anl.gov>
	<AANLkTimp7MskwvD9aFy5qBKvR2--cJJmEKs68W6QFzLV@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1010071604590.1793@wozniak-desktop.mcs.anl.gov>


Yes, but I think that 1.0 is a release candidate- right?  New material 
will go into trunk until we branch for 1.1 .

On Thu, 7 Oct 2010, Sarah Kenny wrote:

> sounds good.
>
> just want to verify (sorry to beat this to death):
>
> so, we alias the 'main guide' to 0.9 (as suggested) and any changes we make
> will go to the doc in branches/1.0 which will then be the new alias for the
> 'main guide' once we do the release.
>
>
>
> On Thu, Oct 7, 2010 at 3:45 PM, Justin M Wozniak <wozniak at mcs.anl.gov>wrote:
>
>>
>> Yup.
>>
>>
>> On Thu, 7 Oct 2010, Mihael Hategan wrote:
>>
>>  To summarize:
>>>
>>> There is a 1-1 mapping between branches and docs. One of the branches
>>> (corresponding to the current release) gets linked from "main" (i.e.
>>> main docs are the docs for the current release). So:
>>>
>>> branches/0.9 <-> 0.9 docs
>>> banches/0.8 <-> 0.8 docs
>>> trunk <-> trunk docs
>>>
>>> If current release is 0.9, then main docs = 0.9 docs.
>>>
>>> On Thu, 2010-10-07 at 15:30 -0500, Justin M Wozniak wrote:
>>>
>>>> I meant release branch.  The valid branches could be hard-coded into the
>>>> update.sh script.  The main guide would be the doc associated with the
>>>> current version.  So right now, "main guide" would be aliased to 0.9 .
>>>>
>>>> On Thu, 7 Oct 2010, Sarah Kenny wrote:
>>>>
>>>>  was thinking of what justin said, "I propose we have one web site but
>>>>> multiple docs/guides directories, all accessible from the docs/index.php
>>>>> page.  Each of these would be associated with a branch"
>>>>>
>>>>> i was assuming that whatever branch(es) these were associated with, that
>>>>> doc
>>>>> would somehow need to make its way to a main guide that we are pointing
>>>>> users to.
>>>>>
>>>>> ?
>>>>>
>>>>>
>>>>> On Thu, Oct 7, 2010 at 3:17 PM, Mihael Hategan <hategan at mcs.anl.gov>
>>>>> wrote:
>>>>>
>>>>>  On Thu, 2010-10-07 at 14:55 -0500, Sarah Kenny wrote:
>>>>>>
>>>>>>> so, in that case the 'main user doc' would be something *like*
>>>>>>> http://www.ci.uchicago.edu/swift/docs10/index.php ?
>>>>>>>
>>>>>>> and THAT would include the updates from all the current branches
>>>>>>>
>>>>>>
>>>>>> define "all current branches". We have:
>>>>>> 1. Release branches
>>>>>> 2. Trunk
>>>>>> 3. Development branches (which are transient entities and only there to
>>>>>> make trunk's life easier).
>>>>>>
>>>>>>   merged into it once we do a release?
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Oct 7, 2010 at 2:42 PM, Mihael Hategan <hategan at mcs.anl.gov>
>>>>>>> wrote:
>>>>>>>        On Thu, 2010-10-07 at 14:39 -0500, Sarah Kenny wrote:
>>>>>>>     > so, in this scenario, the changes to the doc that exist in
>>>>>>>        each branch
>>>>>>>     > are pushed to the main user doc when we do the release or am
>>>>>>>        i missing
>>>>>>>     > a step here?
>>>>>>>
>>>>>>>
>>>>>>>        That or we really have no "main doc" and instead we link from
>>>>>>>        every
>>>>>>>        release. Though I feel odd about that.
>>>>>>>
>>>>>>>     >
>>>>>>>     > On Thu, Oct 7, 2010 at 2:34 PM, Mihael Hategan
>>>>>>>        <hategan at mcs.anl.gov>
>>>>>>>     > wrote:
>>>>>>>     >
>>>>>>>     >         On Thu, 2010-10-07 at 14:28 -0500, Justin M Wozniak
>>>>>>>        wrote:
>>>>>>>     >      > On Thu, 7 Oct 2010, Sarah Kenny wrote:
>>>>>>>     >      >
>>>>>>>     >      >> On Thu, Oct 7, 2010 at 1:08 PM, Mihael Hategan
>>>>>>>     >         <hategan at mcs.anl.gov> wrote:
>>>>>>>     >      >>
>>>>>>>     >      >>> Right. I think 1.0/4.1.7 should go out soon.
>>>>>>>     >      >>
>>>>>>>     >      >> ok, so i guess we should decide what 'soon'
>>>>>>>        means ;) i am
>>>>>>>     >         currently going
>>>>>>>     >      >> thru the old bugs in bugzilla (at least trying
>>>>>>>        to close
>>>>>>>     >         out things that have
>>>>>>>     >      >> been already fixed or are no-longer applicable,
>>>>>>>        etc), but
>>>>>>>     >         perhaps it would
>>>>>>>     >      >> be good to determine if there are bigger issues
>>>>>>>        outside of
>>>>>>>     >         that that still
>>>>>>>     >      >> need to be dealt with before we can put what
>>>>>>>        we've got
>>>>>>>     >         into a stable release
>>>>>>>     >      >> and determine a time-frame...anything come to
>>>>>>>        mind?
>>>>>>>     >      >>
>>>>>>>     >      >> as far as documentation...does it make sense for
>>>>>>>        each
>>>>>>>     >         branch to have a full
>>>>>>>     >      >> copy of /ci/www/projects/swift under it which
>>>>>>>        can then be
>>>>>>>     >         merged with the
>>>>>>>     >      >> main/live copy whenever the code is merged?
>>>>>>>        admittedly, i
>>>>>>>     >         know nothing about
>>>>>>>     >      >> docbook, but from the standpoint of updating and
>>>>>>>        merging
>>>>>>>     >         this seems to make
>>>>>>>     >      >> sense to me (though feel free to suggest another
>>>>>>>        way :)
>>>>>>>     >      >>
>>>>>>>     >      >> ~sk
>>>>>>>     >      >
>>>>>>>     >      > I was looking at the update.sh script earlier
>>>>>>>        today- I
>>>>>>>     >         propose we have one
>>>>>>>     >      > web site but multiple docs/guides directories, all
>>>>>>>     >         accessible from the
>>>>>>>     >      > docs/index.php page.  Each of these would be
>>>>>>>        associated with
>>>>>>>     >         a branch.
>>>>>>>     >      > So, similar to the existing "Historical" section
>>>>>>>        but for
>>>>>>>     >         "future" branches
>>>>>>>     >      > as well.  That would take a small modification to
>>>>>>>        the
>>>>>>>     >         update.sh script and
>>>>>>>     >      > manual modification of the docs/index.php page for
>>>>>>>        each
>>>>>>>     >         version number.
>>>>>>>     >      >
>>>>>>>     >      > We may also want to have the feature changes (past
>>>>>>>        and
>>>>>>>     >         future version
>>>>>>>     >      > numbers) available on that page but I think those
>>>>>>>        can be
>>>>>>>     >         plain text.
>>>>>>>     >      > These could be pulled directly from SVN as well.
>>>>>>>     >      >
>>>>>>>     >
>>>>>>>     >
>>>>>>>     >         I agree. I generally believe that documentation
>>>>>>>        should be kept
>>>>>>>     >         in sync
>>>>>>>     >         with releases (and I also think that the effort of
>>>>>>>        doing so is
>>>>>>>     >         minimal).
>>>>>>>     >
>>>>>>>     >
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>> --
>> Justin M Wozniak
>>
>

-- 
Justin M Wozniak


From wilde at mcs.anl.gov  Thu Oct  7 16:16:57 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 7 Oct 2010 15:16:57 -0600 (GMT-06:00)
Subject: [Swift-devel] swift versions
In-Reply-To: <209692248.851361286486184778.JavaMail.root@zimbra.anl.gov>
Message-ID: <1625742833.851511286486217226.JavaMail.root@zimbra.anl.gov>


----- "Mihael Hategan" <hategan at mcs.anl.gov> wrote:


> That or we really have no "main doc" and instead we link from every
> release. Though I feel odd about that.

I feel this is the way to go - each release has a doc tree. Not every release needs to have its doc tree remain on the web forever, though.  At least Trunk, the latest stable release branch, and any branches that need doc changes to make them usable (eg, possibly things like the "Fast" branch).

- Mike


From wilde at mcs.anl.gov  Thu Oct  7 16:21:01 2010
From: wilde at mcs.anl.gov (wilde at mcs.anl.gov)
Date: Thu, 7 Oct 2010 15:21:01 -0600 (GMT-06:00)
Subject: [Swift-devel] swift versions
In-Reply-To: <946712442.851791286486436981.JavaMail.root@zimbra.anl.gov>
Message-ID: <1825324444.851841286486461139.JavaMail.root@zimbra.anl.gov>


----- "Sarah Kenny" <skenny at uchicago.edu> wrote: 
> was thinking of what justin said, "I propose we have one web site but multiple docs/guides directories, all accessible from the docs/index.php page. Each of these would be associated with a branch" 
> 
> i was assuming that whatever branch(es) these were associated with, that doc would somehow need to make its way to a main guide that we are pointing users to. 

One way to organize this is that the Documentation page be organized like: 

* Latest stable release 
* Development trunk 
* Special development branches 
* Older stable release 

And each of these is a replica of the current Documentation page, cleaned up to remove obsolete content. 

- Mike 

> 
> ? 
> 
> 
> 
> On Thu, Oct 7, 2010 at 3:17 PM, Mihael Hategan < hategan at mcs.anl.gov > wrote: 
> 


> On Thu, 2010-10-07 at 14:55 -0500, Sarah Kenny wrote: 
> > so, in that case the 'main user doc' would be something *like* 
> > http://www.ci.uchicago.edu/swift/docs10/index.php ? 
> > 
> > and THAT would include the updates from all the current branches 
> 
> define "all current branches". We have: 
> 1. Release branches 
> 2. Trunk 
> 3. Development branches (which are transient entities and only there to 
> make trunk's life easier). 
> 


> 
> > merged into it once we do a release? 
> > 
> > 
> > On Thu, Oct 7, 2010 at 2:42 PM, Mihael Hategan < hategan at mcs.anl.gov > 
> > wrote: 
> > On Thu, 2010-10-07 at 14:39 -0500, Sarah Kenny wrote: 
> > > so, in this scenario, the changes to the doc that exist in 
> > each branch 
> > > are pushed to the main user doc when we do the release or am 
> > i missing 
> > > a step here? 
> > 
> > 
> > That or we really have no "main doc" and instead we link from 
> > every 
> > release. Though I feel odd about that. 
> > 
> > > 
> > > On Thu, Oct 7, 2010 at 2:34 PM, Mihael Hategan 
> > < hategan at mcs.anl.gov > 
> > > wrote: 
> > > 
> > > On Thu, 2010-10-07 at 14:28 -0500, Justin M Wozniak 
> > wrote: 
> > > > On Thu, 7 Oct 2010, Sarah Kenny wrote: 
> > > > 
> > > > > On Thu, Oct 7, 2010 at 1:08 PM, Mihael Hategan 
> > > < hategan at mcs.anl.gov > wrote: 
> > > > > 
> > > > >> Right. I think 1.0/4.1.7 should go out soon. 
> > > > > 
> > > > > ok, so i guess we should decide what 'soon' 
> > means ;) i am 
> > > currently going 
> > > > > thru the old bugs in bugzilla (at least trying 
> > to close 
> > > out things that have 
> > > > > been already fixed or are no-longer applicable, 
> > etc), but 
> > > perhaps it would 
> > > > > be good to determine if there are bigger issues 
> > outside of 
> > > that that still 
> > > > > need to be dealt with before we can put what 
> > we've got 
> > > into a stable release 
> > > > > and determine a time-frame...anything come to 
> > mind? 
> > > > > 
> > > > > as far as documentation...does it make sense for 
> > each 
> > > branch to have a full 
> > > > > copy of /ci/www/projects/swift under it which 
> > can then be 
> > > merged with the 
> > > > > main/live copy whenever the code is merged? 
> > admittedly, i 
> > > know nothing about 
> > > > > docbook, but from the standpoint of updating and 
> > merging 
> > > this seems to make 
> > > > > sense to me (though feel free to suggest another 
> > way :) 
> > > > > 
> > > > > ~sk 
> > > > 
> > > > I was looking at the update.sh script earlier 
> > today- I 
> > > propose we have one 
> > > > web site but multiple docs/guides directories, all 
> > > accessible from the 
> > > > docs/index.php page. Each of these would be 
> > associated with 
> > > a branch. 
> > > > So, similar to the existing "Historical" section 
> > but for 
> > > "future" branches 
> > > > as well. That would take a small modification to 
> > the 
> > > update.sh script and 
> > > > manual modification of the docs/index.php page for 
> > each 
> > > version number. 
> > > > 
> > > > We may also want to have the feature changes (past 
> > and 
> > > future version 
> > > > numbers) available on that page but I think those 
> > can be 
> > > plain text. 
> > > > These could be pulled directly from SVN as well. 
> > > > 
> > > 
> > > 
> > > I agree. I generally believe that documentation 
> > should be kept 
> > > in sync 
> > > with releases (and I also think that the effort of 
> > doing so is 
> > > minimal). 
> > > 
> > > 
> > 
> > 
> > 
> > 
> 
> 
> 
> 
> _______________________________________________ Swift-devel mailing list Swift-devel at ci.uchicago.edu http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel 

-- 
Michael Wilde 
Computation Institute, University of Chicago 
Mathematics and Computer Science Division 
Argonne National Laboratory 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101007/98839e1d/attachment.html>

From wilde at mcs.anl.gov  Thu Oct  7 16:26:10 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 7 Oct 2010 15:26:10 -0600 (GMT-06:00)
Subject: [Swift-devel] swift versions
In-Reply-To: <alpine.DEB.2.00.1010071604590.1793@wozniak-desktop.mcs.anl.gov>
Message-ID: <383110182.852171286486770000.JavaMail.root@zimbra.anl.gov>


----- "Justin M Wozniak" <wozniak at mcs.anl.gov> wrote:

> Yes, but I think that 1.0 is a release candidate- right?  New material
> 
> will go into trunk until we branch for 1.1 .

Justin, can you clarify this?  I think we want to take most or all of whats in the trunk *now*, and branch that for 1.0.  That has not been done yet, has it?

If not, lets discuss when to make the branch, and especially how to test.

If its already been done, lets make sure we have all we want/need from trunk in the release branch.

I'm eager to see a 1.0 very soon.

- Mike

> 
> On Thu, 7 Oct 2010, Sarah Kenny wrote:
> 
> > sounds good.
> >
> > just want to verify (sorry to beat this to death):
> >
> > so, we alias the 'main guide' to 0.9 (as suggested) and any changes
> we make
> > will go to the doc in branches/1.0 which will then be the new alias
> for the
> > 'main guide' once we do the release.
> >
> >
> >
> > On Thu, Oct 7, 2010 at 3:45 PM, Justin M Wozniak
> <wozniak at mcs.anl.gov>wrote:
> >
> >>
> >> Yup.
> >>
> >>
> >> On Thu, 7 Oct 2010, Mihael Hategan wrote:
> >>
> >>  To summarize:
> >>>
> >>> There is a 1-1 mapping between branches and docs. One of the
> branches
> >>> (corresponding to the current release) gets linked from "main"
> (i.e.
> >>> main docs are the docs for the current release). So:
> >>>
> >>> branches/0.9 <-> 0.9 docs
> >>> banches/0.8 <-> 0.8 docs
> >>> trunk <-> trunk docs
> >>>
> >>> If current release is 0.9, then main docs = 0.9 docs.
> >>>
> >>> On Thu, 2010-10-07 at 15:30 -0500, Justin M Wozniak wrote:
> >>>
> >>>> I meant release branch.  The valid branches could be hard-coded
> into the
> >>>> update.sh script.  The main guide would be the doc associated
> with the
> >>>> current version.  So right now, "main guide" would be aliased to
> 0.9 .
> >>>>
> >>>> On Thu, 7 Oct 2010, Sarah Kenny wrote:
> >>>>
> >>>>  was thinking of what justin said, "I propose we have one web
> site but
> >>>>> multiple docs/guides directories, all accessible from the
> docs/index.php
> >>>>> page.  Each of these would be associated with a branch"
> >>>>>
> >>>>> i was assuming that whatever branch(es) these were associated
> with, that
> >>>>> doc
> >>>>> would somehow need to make its way to a main guide that we are
> pointing
> >>>>> users to.
> >>>>>
> >>>>> ?
> >>>>>
> >>>>>
> >>>>> On Thu, Oct 7, 2010 at 3:17 PM, Mihael Hategan
> <hategan at mcs.anl.gov>
> >>>>> wrote:
> >>>>>
> >>>>>  On Thu, 2010-10-07 at 14:55 -0500, Sarah Kenny wrote:
> >>>>>>
> >>>>>>> so, in that case the 'main user doc' would be something
> *like*
> >>>>>>> http://www.ci.uchicago.edu/swift/docs10/index.php ?
> >>>>>>>
> >>>>>>> and THAT would include the updates from all the current
> branches
> >>>>>>>
> >>>>>>
> >>>>>> define "all current branches". We have:
> >>>>>> 1. Release branches
> >>>>>> 2. Trunk
> >>>>>> 3. Development branches (which are transient entities and only
> there to
> >>>>>> make trunk's life easier).
> >>>>>>
> >>>>>>   merged into it once we do a release?
> >>>>>>>
> >>>>>>>
> >>>>>>> On Thu, Oct 7, 2010 at 2:42 PM, Mihael Hategan
> <hategan at mcs.anl.gov>
> >>>>>>> wrote:
> >>>>>>>        On Thu, 2010-10-07 at 14:39 -0500, Sarah Kenny wrote:
> >>>>>>>     > so, in this scenario, the changes to the doc that exist
> in
> >>>>>>>        each branch
> >>>>>>>     > are pushed to the main user doc when we do the release
> or am
> >>>>>>>        i missing
> >>>>>>>     > a step here?
> >>>>>>>
> >>>>>>>
> >>>>>>>        That or we really have no "main doc" and instead we
> link from
> >>>>>>>        every
> >>>>>>>        release. Though I feel odd about that.
> >>>>>>>
> >>>>>>>     >
> >>>>>>>     > On Thu, Oct 7, 2010 at 2:34 PM, Mihael Hategan
> >>>>>>>        <hategan at mcs.anl.gov>
> >>>>>>>     > wrote:
> >>>>>>>     >
> >>>>>>>     >         On Thu, 2010-10-07 at 14:28 -0500, Justin M
> Wozniak
> >>>>>>>        wrote:
> >>>>>>>     >      > On Thu, 7 Oct 2010, Sarah Kenny wrote:
> >>>>>>>     >      >
> >>>>>>>     >      >> On Thu, Oct 7, 2010 at 1:08 PM, Mihael Hategan
> >>>>>>>     >         <hategan at mcs.anl.gov> wrote:
> >>>>>>>     >      >>
> >>>>>>>     >      >>> Right. I think 1.0/4.1.7 should go out soon.
> >>>>>>>     >      >>
> >>>>>>>     >      >> ok, so i guess we should decide what 'soon'
> >>>>>>>        means ;) i am
> >>>>>>>     >         currently going
> >>>>>>>     >      >> thru the old bugs in bugzilla (at least trying
> >>>>>>>        to close
> >>>>>>>     >         out things that have
> >>>>>>>     >      >> been already fixed or are no-longer applicable,
> >>>>>>>        etc), but
> >>>>>>>     >         perhaps it would
> >>>>>>>     >      >> be good to determine if there are bigger issues
> >>>>>>>        outside of
> >>>>>>>     >         that that still
> >>>>>>>     >      >> need to be dealt with before we can put what
> >>>>>>>        we've got
> >>>>>>>     >         into a stable release
> >>>>>>>     >      >> and determine a time-frame...anything come to
> >>>>>>>        mind?
> >>>>>>>     >      >>
> >>>>>>>     >      >> as far as documentation...does it make sense
> for
> >>>>>>>        each
> >>>>>>>     >         branch to have a full
> >>>>>>>     >      >> copy of /ci/www/projects/swift under it which
> >>>>>>>        can then be
> >>>>>>>     >         merged with the
> >>>>>>>     >      >> main/live copy whenever the code is merged?
> >>>>>>>        admittedly, i
> >>>>>>>     >         know nothing about
> >>>>>>>     >      >> docbook, but from the standpoint of updating
> and
> >>>>>>>        merging
> >>>>>>>     >         this seems to make
> >>>>>>>     >      >> sense to me (though feel free to suggest
> another
> >>>>>>>        way :)
> >>>>>>>     >      >>
> >>>>>>>     >      >> ~sk
> >>>>>>>     >      >
> >>>>>>>     >      > I was looking at the update.sh script earlier
> >>>>>>>        today- I
> >>>>>>>     >         propose we have one
> >>>>>>>     >      > web site but multiple docs/guides directories,
> all
> >>>>>>>     >         accessible from the
> >>>>>>>     >      > docs/index.php page.  Each of these would be
> >>>>>>>        associated with
> >>>>>>>     >         a branch.
> >>>>>>>     >      > So, similar to the existing "Historical" section
> >>>>>>>        but for
> >>>>>>>     >         "future" branches
> >>>>>>>     >      > as well.  That would take a small modification
> to
> >>>>>>>        the
> >>>>>>>     >         update.sh script and
> >>>>>>>     >      > manual modification of the docs/index.php page
> for
> >>>>>>>        each
> >>>>>>>     >         version number.
> >>>>>>>     >      >
> >>>>>>>     >      > We may also want to have the feature changes
> (past
> >>>>>>>        and
> >>>>>>>     >         future version
> >>>>>>>     >      > numbers) available on that page but I think
> those
> >>>>>>>        can be
> >>>>>>>     >         plain text.
> >>>>>>>     >      > These could be pulled directly from SVN as well.
> >>>>>>>     >      >
> >>>>>>>     >
> >>>>>>>     >
> >>>>>>>     >         I agree. I generally believe that documentation
> >>>>>>>        should be kept
> >>>>>>>     >         in sync
> >>>>>>>     >         with releases (and I also think that the effort
> of
> >>>>>>>        doing so is
> >>>>>>>     >         minimal).
> >>>>>>>     >
> >>>>>>>     >
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >> --
> >> Justin M Wozniak
> >>
> >
> 
> -- 
> Justin M Wozniak
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From wilde at mcs.anl.gov  Thu Oct  7 16:53:51 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 7 Oct 2010 15:53:51 -0600 (GMT-06:00)
Subject: [Swift-devel] swift versions
In-Reply-To: <alpine.DEB.2.00.1010071419540.1793@wozniak-desktop.mcs.anl.gov>
Message-ID: <413556098.853971286488431051.JavaMail.root@zimbra.anl.gov>

Justin, in doing this, can you also make it easy through the build process or some similar mechanism to push the doc revisions in a checked out tree to your local public_html directory, so that you can readily proofread local doc changes and send them out for review and comment?

- Mike

----- "Justin M Wozniak" <wozniak at mcs.anl.gov> wrote:

> On Thu, 7 Oct 2010, Sarah Kenny wrote:
> 
> > On Thu, Oct 7, 2010 at 1:08 PM, Mihael Hategan <hategan at mcs.anl.gov>
> wrote:
> >
> >> Right. I think 1.0/4.1.7 should go out soon.
> >
> > ok, so i guess we should decide what 'soon' means ;) i am currently
> going
> > thru the old bugs in bugzilla (at least trying to close out things
> that have
> > been already fixed or are no-longer applicable, etc), but perhaps it
> would
> > be good to determine if there are bigger issues outside of that that
> still
> > need to be dealt with before we can put what we've got into a stable
> release
> > and determine a time-frame...anything come to mind?
> >
> > as far as documentation...does it make sense for each branch to have
> a full
> > copy of /ci/www/projects/swift under it which can then be merged
> with the
> > main/live copy whenever the code is merged? admittedly, i know
> nothing about
> > docbook, but from the standpoint of updating and merging this seems
> to make
> > sense to me (though feel free to suggest another way :)
> >
> > ~sk
> 
> I was looking at the update.sh script earlier today- I propose we have
> one 
> web site but multiple docs/guides directories, all accessible from the
> 
> docs/index.php page.  Each of these would be associated with a branch.
> 
> So, similar to the existing "Historical" section but for "future"
> branches 
> as well.  That would take a small modification to the update.sh script
> and 
> manual modification of the docs/index.php page for each version
> number.
> 
> We may also want to have the feature changes (past and future version
> 
> numbers) available on that page but I think those can be plain text. 
> These could be pulled directly from SVN as well.
> 
> -- 
> Justin M Wozniak
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From wilde at mcs.anl.gov  Thu Oct  7 17:03:21 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 7 Oct 2010 16:03:21 -0600 (GMT-06:00)
Subject: [Swift-devel] swift versions
In-Reply-To: <AANLkTimGXuDPPbXoDDupsnWMaTREdBEuJ9rugVWcRrDQ@mail.gmail.com>
Message-ID: <1323987572.854551286489001239.JavaMail.root@zimbra.anl.gov>

Hi Sarah, 

Its great that you are cleaning out bugzilla! I'm eager to get it back in general use as our team-wide "to do" tracker. Or at least to try that method and see if it works well for us. 

I have not been seeing any emails from bugzilla, though. Can you see how to configure it so that I get emails on all changes? (same for anyone else on the team who wants to see "everything" - I suspect Justin and Mihael might as well). 

Thanks! 

Mike 




----- "Sarah Kenny" <skenny at uchicago.edu> wrote: 
> 
> On Thu, Oct 7, 2010 at 1:08 PM, Mihael Hategan < hategan at mcs.anl.gov > wrote: 
> 

Right. I think 1.0/4.1.7 should go out soon. 

> ok, so i guess we should decide what 'soon' means ;) i am currently going thru the old bugs in bugzilla (at least trying to close out things that have been already fixed or are no-longer applicable, etc), but perhaps it would be good to determine if there are bigger issues outside of that that still need to be dealt with before we can put what we've got into a stable release and determine a time-frame...anything come to mind? 
> 
> as far as documentation...does it make sense for each branch to have a full copy of /ci/www/projects/swift under it which can then be merged with the main/live copy whenever the code is merged? admittedly, i know nothing about docbook, but from the standpoint of updating and merging this seems to make sense to me (though feel free to suggest another way :) 
> 
> ~sk 
> 
> 

And then 1.1/4.1.8 next. 6 
> months was just a guess at how much it will take us to get to 1.1 and is 
> by no means what I think should happen. But my suspicion is that it will 
> take on the order of months. 
> 



> On Thu, 2010-10-07 at 12:51 -0500, Justin M Wozniak wrote: 
> > Yeah, I would like to get the branches (and branched manuals) formed on 
> > the short term. 
> > 
> > On Thu, 7 Oct 2010, Sarah Kenny wrote: 
> > 
> > > does the 6 month release time that mihael alluded to seem doable? that is, 
> > > do you think we should shoot for the next release being around the 1st week 
> > > of april 2011?...during which time we would phase out clustering (provided 
> > > we don't get anyone on the user list saying they need it...which i sincerely 
> > > doubt). 
> > > 
> > > ~sk 
> > > 
> > > 
> > > On Thu, Oct 7, 2010 at 11:04 AM, Justin M Wozniak < wozniak at mcs.anl.gov >wrote: 
> > > 
> > >> On Tue, 5 Oct 2010, Mihael Hategan wrote: 
> > >> 
> > >> On Tue, 2010-10-05 at 19:40 -0500, Sarah Kenny wrote: 
> > >>> 
> > >>>> ok, just want to make sure i fully understand the state of the code: 
> > >>>> 
> > >>>> these are the most current/stable versions of swift & cog (that which 
> > >>>> we tell users to download): 
> > >>>> 
> > >>>> swift: branch/1.0 
> > >>>> ( https://trac.ci.uchicago.edu/swift/browser/branches/1.0 ) 
> > >>>> cog: branches/4.1.7 
> > >>>> ( http://cogkit.svn.sourceforge.net/viewvc/cogkit/branches/4.1.7/ ) 
> > >>>> 
> > >>>> these are the development (less stable) versions of the code that we 
> > >>>> should be committing changes to: 
> > >>>> 
> > >>>> swift: trunk/ ( https://trac.ci.uchicago.edu/swift/browser/trunk ) 
> > >>>> cog: trunk/current 
> > >>>> ( http://cogkit.svn.sourceforge.net/viewvc/cogkit/trunk/current/ ) 
> > >>>> 
> > >>>> assuming that's correct... 
> > >>>> 
> > >>>> for swift, what are the other/multiple directories under branches/ 
> > >>>> that do not appear to be releases or release candidates? 
> > >>>> is the intent, upon the next swift release (whenever that may be) to 
> > >>>> update swift branch/1.0 with the changes that have been committed to 
> > >>>> trunk/ ? 
> > >>>> 
> > >>> 
> > >>> Right. There are two reasons for doing branches: 
> > >>> 1. if some work divergent from trunk is needed 
> > >>> 2. to stabilize for a release 
> > >>> 
> > >>> For (1), if thinks work out OK, you eventually merge them back into 
> > >>> trunk. 
> > >>> 
> > >>> (2)s also get merged back into trunk in one way or another, since they 
> > >>> may contain bug fixes. 
> > >>> 
> > >>> _______________________________________________ 
> > >>> Swift-devel mailing list 
> > >>> Swift-devel at ci.uchicago.edu 
> > >>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel 
> > >>> 
> > >> 
> > >> So for Swift 1.1, can we have a CoG 4.1.8? 
> > >> 
> > >> -- 
> > >> Justin M Wozniak 
> > >> 
> > > 
> > 
> 
> 
> 
> 
> _______________________________________________ Swift-devel mailing list Swift-devel at ci.uchicago.edu http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel 

-- 
Michael Wilde 
Computation Institute, University of Chicago 
Mathematics and Computer Science Division 
Argonne National Laboratory 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101007/8f6197ab/attachment.html>

From skenny at uchicago.edu  Thu Oct  7 17:07:17 2010
From: skenny at uchicago.edu (Sarah Kenny)
Date: Thu, 7 Oct 2010 17:07:17 -0500
Subject: [Swift-devel] swift versions
In-Reply-To: <1323987572.854551286489001239.JavaMail.root@zimbra.anl.gov>
References: <AANLkTimGXuDPPbXoDDupsnWMaTREdBEuJ9rugVWcRrDQ@mail.gmail.com>
	<1323987572.854551286489001239.JavaMail.root@zimbra.anl.gov>
Message-ID: <AANLkTi=Ttgm5SDf-HiWn20zJOr-4LwRuNsH7itf0gRJc@mail.gmail.com>

i'm getting the emails (coming thru to swift-devel from bugzilla)...is
anyone else getting them?

~sk

On Thu, Oct 7, 2010 at 5:03 PM, Michael Wilde <wilde at mcs.anl.gov> wrote:

> Hi Sarah,
>
> Its great that you are cleaning out bugzilla! I'm eager to get it back in
> general use as our team-wide "to do" tracker. Or at least to try that method
> and see if it works well for us.
>
> I have not been seeing any emails from bugzilla, though. Can you see how to
> configure it so that I get emails on all changes? (same for anyone else on
> the team who wants to see "everything" - I suspect Justin and Mihael might
> as well).
>
> Thanks!
>
> Mike
>
>
>
>
>
> ----- "Sarah Kenny" <skenny at uchicago.edu> wrote:
> >
> > On Thu, Oct 7, 2010 at 1:08 PM, Mihael Hategan <hategan at mcs.anl.gov>wrote:
> >
>>
>> Right. I think 1.0/4.1.7 should go out soon.
>
>
> > ok, so i guess we should decide what 'soon' means ;) i am currently going
> thru the old bugs in bugzilla (at least trying to close out things that have
> been already fixed or are no-longer applicable, etc), but perhaps it would
> be good to determine if there are bigger issues outside of that that still
> need to be dealt with before we can put what we've got into a stable release
> and determine a time-frame...anything come to mind?
> >
> > as far as documentation...does it make sense for each branch to have a
> full copy of /ci/www/projects/swift under it which can then be merged with
> the main/live copy whenever the code is merged? admittedly, i know nothing
> about docbook, but from the standpoint of updating and merging this seems to
> make sense to me (though feel free to suggest another way :)
> >
> > ~sk
> >
> >
>
>> And then 1.1/4.1.8 next. 6
>> > months was just a guess at how much it will take us to get to 1.1 and is
>> > by no means what I think should happen. But my suspicion is that it will
>> > take on the order of months.
>> >
>>
>> > On Thu, 2010-10-07 at 12:51 -0500, Justin M Wozniak wrote:
>> > > Yeah, I would like to get the branches (and branched manuals) formed
>> on
>> > > the short term.
>> > >
>> > > On Thu, 7 Oct 2010, Sarah Kenny wrote:
>> > >
>> > > > does the 6 month release time that mihael alluded to seem doable?
>> that is,
>> > > > do you think we should shoot for the next release being around the
>> 1st week
>> > > > of april 2011?...during which time we would phase out clustering
>> (provided
>> > > > we don't get anyone on the user list saying they need it...which i
>> sincerely
>> > > > doubt).
>> > > >
>> > > > ~sk
>> > > >
>> > > >
>> > > > On Thu, Oct 7, 2010 at 11:04 AM, Justin M Wozniak <
>> wozniak at mcs.anl.gov>wrote:
>> > > >
>> > > >> On Tue, 5 Oct 2010, Mihael Hategan wrote:
>> > > >>
>> > > >>  On Tue, 2010-10-05 at 19:40 -0500, Sarah Kenny wrote:
>> > > >>>
>> > > >>>> ok, just want to make sure i fully understand the state of the
>> code:
>> > > >>>>
>> > > >>>> these are the most current/stable versions of swift & cog (that
>> which
>> > > >>>> we tell users to download):
>> > > >>>>
>> > > >>>> swift: branch/1.0
>> > > >>>> (https://trac.ci.uchicago.edu/swift/browser/branches/1.0)
>> > > >>>> cog: branches/4.1.7
>> > > >>>> (http://cogkit.svn.sourceforge.net/viewvc/cogkit/branches/4.1.7/
>> )
>> > > >>>>
>> > > >>>> these are the development (less stable) versions of the code that
>> we
>> > > >>>> should be committing changes to:
>> > > >>>>
>> > > >>>> swift: trunk/ (https://trac.ci.uchicago.edu/swift/browser/trunk)
>> > > >>>> cog: trunk/current
>> > > >>>> (http://cogkit.svn.sourceforge.net/viewvc/cogkit/trunk/current/)
>> > > >>>>
>> > > >>>> assuming that's correct...
>> > > >>>>
>> > > >>>> for swift, what are the other/multiple directories under
>> branches/
>> > > >>>> that do not appear to be releases or release candidates?
>> > > >>>> is the intent, upon the next swift release (whenever that may be)
>> to
>> > > >>>> update swift branch/1.0 with the changes that have been committed
>> to
>> > > >>>> trunk/ ?
>> > > >>>>
>> > > >>>
>> > > >>> Right. There are two reasons for doing branches:
>> > > >>> 1. if some work divergent from trunk is needed
>> > > >>> 2. to stabilize for a release
>> > > >>>
>> > > >>> For (1), if thinks work out OK, you eventually merge them back
>> into
>> > > >>> trunk.
>> > > >>>
>> > > >>> (2)s also get merged back into trunk in one way or another, since
>> they
>> > > >>> may contain bug fixes.
>> > > >>>
>> > > >>> _______________________________________________
>> > > >>> Swift-devel mailing list
>> > > >>> Swift-devel at ci.uchicago.edu
>> > > >>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>> > > >>>
>> > > >>
>> > > >> So for Swift 1.1, can we have a CoG 4.1.8?
>> > > >>
>> > > >> --
>> > > >> Justin M Wozniak
>> > > >>
>> > > >
>> > >
>> >
>> >
>> >
>>
>
> >
> > _______________________________________________ Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101007/a349373a/attachment.html>

From hategan at mcs.anl.gov  Thu Oct  7 17:05:22 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 07 Oct 2010 15:05:22 -0700
Subject: [Swift-devel] swift versions
In-Reply-To: <383110182.852171286486770000.JavaMail.root@zimbra.anl.gov>
References: <383110182.852171286486770000.JavaMail.root@zimbra.anl.gov>
Message-ID: <1286489122.28173.0.camel@blabla2.none>

On Thu, 2010-10-07 at 15:26 -0600, Michael Wilde wrote:
> ----- "Justin M Wozniak" <wozniak at mcs.anl.gov> wrote:
> 
> > Yes, but I think that 1.0 is a release candidate- right?  New material
> > 
> > will go into trunk until we branch for 1.1 .
> 
> Justin, can you clarify this?  I think we want to take most or all of
> whats in the trunk *now*, and branch that for 1.0.  That has not been
> done yet, has it?

It has. For a long time. We have not yet released branches/1.0.

> 
> If not, lets discuss when to make the branch, and especially how to test.
> 
> If its already been done, lets make sure we have all we want/need from trunk in the release branch.
> 
> I'm eager to see a 1.0 very soon.
> 
> - Mike
> 
> > 
> > On Thu, 7 Oct 2010, Sarah Kenny wrote:
> > 
> > > sounds good.
> > >
> > > just want to verify (sorry to beat this to death):
> > >
> > > so, we alias the 'main guide' to 0.9 (as suggested) and any changes
> > we make
> > > will go to the doc in branches/1.0 which will then be the new alias
> > for the
> > > 'main guide' once we do the release.
> > >
> > >
> > >
> > > On Thu, Oct 7, 2010 at 3:45 PM, Justin M Wozniak
> > <wozniak at mcs.anl.gov>wrote:
> > >
> > >>
> > >> Yup.
> > >>
> > >>
> > >> On Thu, 7 Oct 2010, Mihael Hategan wrote:
> > >>
> > >>  To summarize:
> > >>>
> > >>> There is a 1-1 mapping between branches and docs. One of the
> > branches
> > >>> (corresponding to the current release) gets linked from "main"
> > (i.e.
> > >>> main docs are the docs for the current release). So:
> > >>>
> > >>> branches/0.9 <-> 0.9 docs
> > >>> banches/0.8 <-> 0.8 docs
> > >>> trunk <-> trunk docs
> > >>>
> > >>> If current release is 0.9, then main docs = 0.9 docs.
> > >>>
> > >>> On Thu, 2010-10-07 at 15:30 -0500, Justin M Wozniak wrote:
> > >>>
> > >>>> I meant release branch.  The valid branches could be hard-coded
> > into the
> > >>>> update.sh script.  The main guide would be the doc associated
> > with the
> > >>>> current version.  So right now, "main guide" would be aliased to
> > 0.9 .
> > >>>>
> > >>>> On Thu, 7 Oct 2010, Sarah Kenny wrote:
> > >>>>
> > >>>>  was thinking of what justin said, "I propose we have one web
> > site but
> > >>>>> multiple docs/guides directories, all accessible from the
> > docs/index.php
> > >>>>> page.  Each of these would be associated with a branch"
> > >>>>>
> > >>>>> i was assuming that whatever branch(es) these were associated
> > with, that
> > >>>>> doc
> > >>>>> would somehow need to make its way to a main guide that we are
> > pointing
> > >>>>> users to.
> > >>>>>
> > >>>>> ?
> > >>>>>
> > >>>>>
> > >>>>> On Thu, Oct 7, 2010 at 3:17 PM, Mihael Hategan
> > <hategan at mcs.anl.gov>
> > >>>>> wrote:
> > >>>>>
> > >>>>>  On Thu, 2010-10-07 at 14:55 -0500, Sarah Kenny wrote:
> > >>>>>>
> > >>>>>>> so, in that case the 'main user doc' would be something
> > *like*
> > >>>>>>> http://www.ci.uchicago.edu/swift/docs10/index.php ?
> > >>>>>>>
> > >>>>>>> and THAT would include the updates from all the current
> > branches
> > >>>>>>>
> > >>>>>>
> > >>>>>> define "all current branches". We have:
> > >>>>>> 1. Release branches
> > >>>>>> 2. Trunk
> > >>>>>> 3. Development branches (which are transient entities and only
> > there to
> > >>>>>> make trunk's life easier).
> > >>>>>>
> > >>>>>>   merged into it once we do a release?
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Thu, Oct 7, 2010 at 2:42 PM, Mihael Hategan
> > <hategan at mcs.anl.gov>
> > >>>>>>> wrote:
> > >>>>>>>        On Thu, 2010-10-07 at 14:39 -0500, Sarah Kenny wrote:
> > >>>>>>>     > so, in this scenario, the changes to the doc that exist
> > in
> > >>>>>>>        each branch
> > >>>>>>>     > are pushed to the main user doc when we do the release
> > or am
> > >>>>>>>        i missing
> > >>>>>>>     > a step here?
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>        That or we really have no "main doc" and instead we
> > link from
> > >>>>>>>        every
> > >>>>>>>        release. Though I feel odd about that.
> > >>>>>>>
> > >>>>>>>     >
> > >>>>>>>     > On Thu, Oct 7, 2010 at 2:34 PM, Mihael Hategan
> > >>>>>>>        <hategan at mcs.anl.gov>
> > >>>>>>>     > wrote:
> > >>>>>>>     >
> > >>>>>>>     >         On Thu, 2010-10-07 at 14:28 -0500, Justin M
> > Wozniak
> > >>>>>>>        wrote:
> > >>>>>>>     >      > On Thu, 7 Oct 2010, Sarah Kenny wrote:
> > >>>>>>>     >      >
> > >>>>>>>     >      >> On Thu, Oct 7, 2010 at 1:08 PM, Mihael Hategan
> > >>>>>>>     >         <hategan at mcs.anl.gov> wrote:
> > >>>>>>>     >      >>
> > >>>>>>>     >      >>> Right. I think 1.0/4.1.7 should go out soon.
> > >>>>>>>     >      >>
> > >>>>>>>     >      >> ok, so i guess we should decide what 'soon'
> > >>>>>>>        means ;) i am
> > >>>>>>>     >         currently going
> > >>>>>>>     >      >> thru the old bugs in bugzilla (at least trying
> > >>>>>>>        to close
> > >>>>>>>     >         out things that have
> > >>>>>>>     >      >> been already fixed or are no-longer applicable,
> > >>>>>>>        etc), but
> > >>>>>>>     >         perhaps it would
> > >>>>>>>     >      >> be good to determine if there are bigger issues
> > >>>>>>>        outside of
> > >>>>>>>     >         that that still
> > >>>>>>>     >      >> need to be dealt with before we can put what
> > >>>>>>>        we've got
> > >>>>>>>     >         into a stable release
> > >>>>>>>     >      >> and determine a time-frame...anything come to
> > >>>>>>>        mind?
> > >>>>>>>     >      >>
> > >>>>>>>     >      >> as far as documentation...does it make sense
> > for
> > >>>>>>>        each
> > >>>>>>>     >         branch to have a full
> > >>>>>>>     >      >> copy of /ci/www/projects/swift under it which
> > >>>>>>>        can then be
> > >>>>>>>     >         merged with the
> > >>>>>>>     >      >> main/live copy whenever the code is merged?
> > >>>>>>>        admittedly, i
> > >>>>>>>     >         know nothing about
> > >>>>>>>     >      >> docbook, but from the standpoint of updating
> > and
> > >>>>>>>        merging
> > >>>>>>>     >         this seems to make
> > >>>>>>>     >      >> sense to me (though feel free to suggest
> > another
> > >>>>>>>        way :)
> > >>>>>>>     >      >>
> > >>>>>>>     >      >> ~sk
> > >>>>>>>     >      >
> > >>>>>>>     >      > I was looking at the update.sh script earlier
> > >>>>>>>        today- I
> > >>>>>>>     >         propose we have one
> > >>>>>>>     >      > web site but multiple docs/guides directories,
> > all
> > >>>>>>>     >         accessible from the
> > >>>>>>>     >      > docs/index.php page.  Each of these would be
> > >>>>>>>        associated with
> > >>>>>>>     >         a branch.
> > >>>>>>>     >      > So, similar to the existing "Historical" section
> > >>>>>>>        but for
> > >>>>>>>     >         "future" branches
> > >>>>>>>     >      > as well.  That would take a small modification
> > to
> > >>>>>>>        the
> > >>>>>>>     >         update.sh script and
> > >>>>>>>     >      > manual modification of the docs/index.php page
> > for
> > >>>>>>>        each
> > >>>>>>>     >         version number.
> > >>>>>>>     >      >
> > >>>>>>>     >      > We may also want to have the feature changes
> > (past
> > >>>>>>>        and
> > >>>>>>>     >         future version
> > >>>>>>>     >      > numbers) available on that page but I think
> > those
> > >>>>>>>        can be
> > >>>>>>>     >         plain text.
> > >>>>>>>     >      > These could be pulled directly from SVN as well.
> > >>>>>>>     >      >
> > >>>>>>>     >
> > >>>>>>>     >
> > >>>>>>>     >         I agree. I generally believe that documentation
> > >>>>>>>        should be kept
> > >>>>>>>     >         in sync
> > >>>>>>>     >         with releases (and I also think that the effort
> > of
> > >>>>>>>        doing so is
> > >>>>>>>     >         minimal).
> > >>>>>>>     >
> > >>>>>>>     >
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>>
> > >>>
> > >> --
> > >> Justin M Wozniak
> > >>
> > >
> > 
> > -- 
> > Justin M Wozniak
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> 




From wozniak at mcs.anl.gov  Thu Oct  7 17:08:15 2010
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Thu, 7 Oct 2010 17:08:15 -0500 (CDT)
Subject: [Swift-devel] swift versions
In-Reply-To: <AANLkTi=Ttgm5SDf-HiWn20zJOr-4LwRuNsH7itf0gRJc@mail.gmail.com>
References: <AANLkTimGXuDPPbXoDDupsnWMaTREdBEuJ9rugVWcRrDQ@mail.gmail.com>
	<1323987572.854551286489001239.JavaMail.root@zimbra.anl.gov>
	<AANLkTi=Ttgm5SDf-HiWn20zJOr-4LwRuNsH7itf0gRJc@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1010071708070.1793@wozniak-desktop.mcs.anl.gov>


I'm getting them.

On Thu, 7 Oct 2010, Sarah Kenny wrote:

> i'm getting the emails (coming thru to swift-devel from bugzilla)...is
> anyone else getting them?
>
> ~sk
>
> On Thu, Oct 7, 2010 at 5:03 PM, Michael Wilde <wilde at mcs.anl.gov> wrote:
>
>> Hi Sarah,
>>
>> Its great that you are cleaning out bugzilla! I'm eager to get it back in
>> general use as our team-wide "to do" tracker. Or at least to try that method
>> and see if it works well for us.
>>
>> I have not been seeing any emails from bugzilla, though. Can you see how to
>> configure it so that I get emails on all changes? (same for anyone else on
>> the team who wants to see "everything" - I suspect Justin and Mihael might
>> as well).
>>
>> Thanks!
>>
>> Mike
>>
>>
>>
>>
>>
>> ----- "Sarah Kenny" <skenny at uchicago.edu> wrote:
>>>
>>> On Thu, Oct 7, 2010 at 1:08 PM, Mihael Hategan <hategan at mcs.anl.gov>wrote:
>>>
>>>
>>> Right. I think 1.0/4.1.7 should go out soon.
>>
>>
>>> ok, so i guess we should decide what 'soon' means ;) i am currently going
>> thru the old bugs in bugzilla (at least trying to close out things that have
>> been already fixed or are no-longer applicable, etc), but perhaps it would
>> be good to determine if there are bigger issues outside of that that still
>> need to be dealt with before we can put what we've got into a stable release
>> and determine a time-frame...anything come to mind?
>>>
>>> as far as documentation...does it make sense for each branch to have a
>> full copy of /ci/www/projects/swift under it which can then be merged with
>> the main/live copy whenever the code is merged? admittedly, i know nothing
>> about docbook, but from the standpoint of updating and merging this seems to
>> make sense to me (though feel free to suggest another way :)
>>>
>>> ~sk
>>>
>>>
>>
>>> And then 1.1/4.1.8 next. 6
>>>> months was just a guess at how much it will take us to get to 1.1 and is
>>>> by no means what I think should happen. But my suspicion is that it will
>>>> take on the order of months.
>>>>
>>>
>>>> On Thu, 2010-10-07 at 12:51 -0500, Justin M Wozniak wrote:
>>>>> Yeah, I would like to get the branches (and branched manuals) formed
>>> on
>>>>> the short term.
>>>>>
>>>>> On Thu, 7 Oct 2010, Sarah Kenny wrote:
>>>>>
>>>>>> does the 6 month release time that mihael alluded to seem doable?
>>> that is,
>>>>>> do you think we should shoot for the next release being around the
>>> 1st week
>>>>>> of april 2011?...during which time we would phase out clustering
>>> (provided
>>>>>> we don't get anyone on the user list saying they need it...which i
>>> sincerely
>>>>>> doubt).
>>>>>>
>>>>>> ~sk
>>>>>>
>>>>>>
>>>>>> On Thu, Oct 7, 2010 at 11:04 AM, Justin M Wozniak <
>>> wozniak at mcs.anl.gov>wrote:
>>>>>>
>>>>>>> On Tue, 5 Oct 2010, Mihael Hategan wrote:
>>>>>>>
>>>>>>>  On Tue, 2010-10-05 at 19:40 -0500, Sarah Kenny wrote:
>>>>>>>>
>>>>>>>>> ok, just want to make sure i fully understand the state of the
>>> code:
>>>>>>>>>
>>>>>>>>> these are the most current/stable versions of swift & cog (that
>>> which
>>>>>>>>> we tell users to download):
>>>>>>>>>
>>>>>>>>> swift: branch/1.0
>>>>>>>>> (https://trac.ci.uchicago.edu/swift/browser/branches/1.0)
>>>>>>>>> cog: branches/4.1.7
>>>>>>>>> (http://cogkit.svn.sourceforge.net/viewvc/cogkit/branches/4.1.7/
>>> )
>>>>>>>>>
>>>>>>>>> these are the development (less stable) versions of the code that
>>> we
>>>>>>>>> should be committing changes to:
>>>>>>>>>
>>>>>>>>> swift: trunk/ (https://trac.ci.uchicago.edu/swift/browser/trunk)
>>>>>>>>> cog: trunk/current
>>>>>>>>> (http://cogkit.svn.sourceforge.net/viewvc/cogkit/trunk/current/)
>>>>>>>>>
>>>>>>>>> assuming that's correct...
>>>>>>>>>
>>>>>>>>> for swift, what are the other/multiple directories under
>>> branches/
>>>>>>>>> that do not appear to be releases or release candidates?
>>>>>>>>> is the intent, upon the next swift release (whenever that may be)
>>> to
>>>>>>>>> update swift branch/1.0 with the changes that have been committed
>>> to
>>>>>>>>> trunk/ ?
>>>>>>>>>
>>>>>>>>
>>>>>>>> Right. There are two reasons for doing branches:
>>>>>>>> 1. if some work divergent from trunk is needed
>>>>>>>> 2. to stabilize for a release
>>>>>>>>
>>>>>>>> For (1), if thinks work out OK, you eventually merge them back
>>> into
>>>>>>>> trunk.
>>>>>>>>
>>>>>>>> (2)s also get merged back into trunk in one way or another, since
>>> they
>>>>>>>> may contain bug fixes.
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Swift-devel mailing list
>>>>>>>> Swift-devel at ci.uchicago.edu
>>>>>>>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>>>>>>>>
>>>>>>>
>>>>>>> So for Swift 1.1, can we have a CoG 4.1.8?
>>>>>>>
>>>>>>> --
>>>>>>> Justin M Wozniak
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>
>>>
>>> _______________________________________________ Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>>
>> --
>> Michael Wilde
>> Computation Institute, University of Chicago
>> Mathematics and Computer Science Division
>> Argonne National Laboratory
>>
>>
>

-- 
Justin M Wozniak


From wilde at mcs.anl.gov  Thu Oct  7 17:12:43 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 7 Oct 2010 16:12:43 -0600 (GMT-06:00)
Subject: [Swift-devel] swift versions
In-Reply-To: <AANLkTi=Ttgm5SDf-HiWn20zJOr-4LwRuNsH7itf0gRJc@mail.gmail.com>
Message-ID: <19298389.855121286489563546.JavaMail.root@zimbra.anl.gov>

Sorry - searching more carefully, I *have* been getting them! 

Thanks, 

- Mike 

----- "Sarah Kenny" <skenny at uchicago.edu> wrote: 
> i'm getting the emails (coming thru to swift-devel from bugzilla)...is anyone else getting them? 
> 
> ~sk 
> 
> 
> On Thu, Oct 7, 2010 at 5:03 PM, Michael Wilde < wilde at mcs.anl.gov > wrote: 
> 



> Hi Sarah, 
> 
> Its great that you are cleaning out bugzilla! I'm eager to get it back in general use as our team-wide "to do" tracker. Or at least to try that method and see if it works well for us. 
> 
> I have not been seeing any emails from bugzilla, though. Can you see how to configure it so that I get emails on all changes? (same for anyone else on the team who wants to see "everything" - I suspect Justin and Mihael might as well). 
> 
> Thanks! 
> 
> Mike 

> 
> 
> 
> 
> ----- "Sarah Kenny" < skenny at uchicago.edu > wrote: 
> > 



> > On Thu, Oct 7, 2010 at 1:08 PM, Mihael Hategan < hategan at mcs.anl.gov > wrote: 
> > 

Right. I think 1.0/4.1.7 should go out soon. 

> > ok, so i guess we should decide what 'soon' means ;) i am currently going thru the old bugs in bugzilla (at least trying to close out things that have been already fixed or are no-longer applicable, etc), but perhaps it would be good to determine if there are bigger issues outside of that that still need to be dealt with before we can put what we've got into a stable release and determine a time-frame...anything come to mind? 
> > 
> > as far as documentation...does it make sense for each branch to have a full copy of /ci/www/projects/swift under it which can then be merged with the main/live copy whenever the code is merged? admittedly, i know nothing about docbook, but from the standpoint of updating and merging this seems to make sense to me (though feel free to suggest another way :) 
> > 
> > ~sk 
> > 
> > 

And then 1.1/4.1.8 next. 6 
> > months was just a guess at how much it will take us to get to 1.1 and is 
> > by no means what I think should happen. But my suspicion is that it will 
> > take on the order of months. 
> > 



> > On Thu, 2010-10-07 at 12:51 -0500, Justin M Wozniak wrote: 
> > > Yeah, I would like to get the branches (and branched manuals) formed on 
> > > the short term. 
> > > 
> > > On Thu, 7 Oct 2010, Sarah Kenny wrote: 
> > > 
> > > > does the 6 month release time that mihael alluded to seem doable? that is, 
> > > > do you think we should shoot for the next release being around the 1st week 
> > > > of april 2011?...during which time we would phase out clustering (provided 
> > > > we don't get anyone on the user list saying they need it...which i sincerely 
> > > > doubt). 
> > > > 
> > > > ~sk 
> > > > 
> > > > 
> > > > On Thu, Oct 7, 2010 at 11:04 AM, Justin M Wozniak < wozniak at mcs.anl.gov >wrote: 
> > > > 
> > > >> On Tue, 5 Oct 2010, Mihael Hategan wrote: 
> > > >> 
> > > >> On Tue, 2010-10-05 at 19:40 -0500, Sarah Kenny wrote: 
> > > >>> 
> > > >>>> ok, just want to make sure i fully understand the state of the code: 
> > > >>>> 
> > > >>>> these are the most current/stable versions of swift & cog (that which 
> > > >>>> we tell users to download): 
> > > >>>> 
> > > >>>> swift: branch/1.0 
> > > >>>> ( https://trac.ci.uchicago.edu/swift/browser/branches/1.0 ) 
> > > >>>> cog: branches/4.1.7 
> > > >>>> ( http://cogkit.svn.sourceforge.net/viewvc/cogkit/branches/4.1.7/ ) 
> > > >>>> 
> > > >>>> these are the development (less stable) versions of the code that we 
> > > >>>> should be committing changes to: 
> > > >>>> 
> > > >>>> swift: trunk/ ( https://trac.ci.uchicago.edu/swift/browser/trunk ) 
> > > >>>> cog: trunk/current 
> > > >>>> ( http://cogkit.svn.sourceforge.net/viewvc/cogkit/trunk/current/ ) 
> > > >>>> 
> > > >>>> assuming that's correct... 
> > > >>>> 
> > > >>>> for swift, what are the other/multiple directories under branches/ 
> > > >>>> that do not appear to be releases or release candidates? 
> > > >>>> is the intent, upon the next swift release (whenever that may be) to 
> > > >>>> update swift branch/1.0 with the changes that have been committed to 
> > > >>>> trunk/ ? 
> > > >>>> 
> > > >>> 
> > > >>> Right. There are two reasons for doing branches: 
> > > >>> 1. if some work divergent from trunk is needed 
> > > >>> 2. to stabilize for a release 
> > > >>> 
> > > >>> For (1), if thinks work out OK, you eventually merge them back into 
> > > >>> trunk. 
> > > >>> 
> > > >>> (2)s also get merged back into trunk in one way or another, since they 
> > > >>> may contain bug fixes. 
> > > >>> 
> > > >>> _______________________________________________ 
> > > >>> Swift-devel mailing list 
> > > >>> Swift-devel at ci.uchicago.edu 
> > > >>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel 
> > > >>> 
> > > >> 
> > > >> So for Swift 1.1, can we have a CoG 4.1.8? 
> > > >> 
> > > >> -- 
> > > >> Justin M Wozniak 
> > > >> 
> > > > 
> > > 
> > 
> > 
> > 
> > 
> > _______________________________________________ Swift-devel mailing list Swift-devel at ci.uchicago.edu http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel 
> 

> -- 
> Michael Wilde 
> Computation Institute, University of Chicago 
> Mathematics and Computer Science Division 
> Argonne National Laboratory 
> 
> 
> 

-- 
Michael Wilde 
Computation Institute, University of Chicago 
Mathematics and Computer Science Division 
Argonne National Laboratory 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101007/62868086/attachment.html>

From bugzilla-daemon at mcs.anl.gov  Thu Oct  7 17:47:55 2010
From: bugzilla-daemon at mcs.anl.gov (bugzilla-daemon at mcs.anl.gov)
Date: Thu,  7 Oct 2010 17:47:55 -0500 (CDT)
Subject: [Swift-devel] [Bug 40] source location indication in execution-time
	error messages
In-Reply-To: <bug-40-21@http.bugzilla.mcs.anl.gov/swift/>
References: <bug-40-21@http.bugzilla.mcs.anl.gov/swift/>
Message-ID: <20101007224755.9BB2E2CA15@wind.mcs.anl.gov>

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=40


skenny <skenny at uchicago.edu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |skenny at uchicago.edu




--- Comment #3 from skenny <skenny at uchicago.edu>  2010-10-07 17:47:55 ---
in the current swift, i seem to get no error if an index is out of bounds, but
it simply hangs indefinitely:

[skenny at communicado]$ swift t1.swift
Swift svn swift-r3497 cog-r2829 (cog modified locally)

RunID: 20101007-1744-m8yu5923
Progress:
Progress:
Progress:
[skenny at communicado]$ cat t1.swift
type messagefile;

type file;

(messagefile t) echo ( string s1 ) {
    app {
        echo s1 stdout=@filename(t);
    }
}

messagefile outfile <"t1out.txt">;

string mystrings[] = ["blah","blah2","blah3"];

outfile = echo( mystrings[3] );

-- 
Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching the reporter.


From wilde at mcs.anl.gov  Thu Oct  7 17:55:02 2010
From: wilde at mcs.anl.gov (wilde at mcs.anl.gov)
Date: Thu, 7 Oct 2010 16:55:02 -0600 (GMT-06:00)
Subject: [Swift-devel] Coaster worker connect problem and control over
	coaster logging?
In-Reply-To: <1793650383.857081286491696387.JavaMail.root@zimbra.anl.gov>
Message-ID: <2068606027.857261286492102885.JavaMail.root@zimbra.anl.gov>

Im debugging a problem I just started seeing in coasters on PADS. Im getting "error code 29" returned from a simple 1-cat job.

Whats happening is that the coaster worker is failing to connect. Its possible I broke it in a recent commit; I hope to know soon. I see this in the worker log:

1286490899.535 DEBUG - Trying 169.254.95.119:35151...
1286490899.571 DEBUG - Connection failed: Connection refused. Trying other addresses
1286490899.571 DEBUG - Trying 172.5.86.5:35151...
1286490899.573 DEBUG - Connection failed: Illegal seek. Trying other addresses
1286490899.573 DEBUG - Trying 192.5.86.5:35151...
1286490899.574 DEBUG - Connection failed: Illegal seek. Trying other addresses
1286490899.574 ERROR - Connection failed for all addresses. Retrying in 1 seconds

A second question here is about control of the worker log. I see the env variable WORKER_LOGGING_ENABLED getting set in the coaster pbs submit file. But as far as I can tell, this will not be picked up by the worker unless its exported.

Has this always been set this way? Is anyone actually *getting* worker logs in their ~/.globus/coasters directory using trunk?

I will look further into this; in the meantime Im forcing TRACE logging on in worker.pl (which is how I finally got the messages above).

Sarah: this is an interesting and challenging case in error reporting. Diagnosing this involves tracking the error from swift stdout to the pbs stderr file (with debug=true in the etc/provider-pbs.properties file) to the coaster worker log (with elevated logging levels).  Lets discuss how the defaults in all these 3 places could be better, and how the relevant files could be better coalesced for the user, and perhaps integrated by some post-processing diagnostic tool.

- Mike

login1$ cat *54.submit
#PBS -S /bin/sh
#PBS -N Block-1007-340807-000000
#PBS -m n
#PBS -l nodes=1
#PBS -l walltime=01:00:00
#PBS -q short
#PBS -o /home/wilde/.globus/scripts/PBS3642641579913160354.submit.stdout
#PBS -e /home/wilde/.globus/scripts/PBS3642641579913160354.submit.stderr
WORKER_LOGGING_ENABLED=true
cd / && /usr/bin/perl /home/wilde/.globus/coasters/cscript4619936640935778716.pl http://169.254.95.119:37300,http://172.5.86.5:37300,http://192.5.86.5:37300 1007-340807-000000 /home/wilde/.globus/coasters
/bin/echo $? >/home/wilde/.globus/scripts/PBS3642641579913160354.submit.exitcode
login1$ 


-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From skenny at uchicago.edu  Thu Oct  7 18:08:17 2010
From: skenny at uchicago.edu (Sarah Kenny)
Date: Thu, 7 Oct 2010 18:08:17 -0500
Subject: [Swift-devel] kickstart?
Message-ID: <AANLkTi=zcZC9OP92zieJmvhy4EGjNs9z_b_pjenQ-sBc@mail.gmail.com>

anyone currently using kickstart?

i ask bcs it's in the doc (and mentioned in one of the bugs) but my group
has never used it on teragrid (not since the days of vds :) also i don't see
references to it on the user list so i was curious if it's still in use?

~sk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101007/eedcc50d/attachment.html>

From wilde at mcs.anl.gov  Thu Oct  7 18:27:06 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 7 Oct 2010 17:27:06 -0600 (GMT-06:00)
Subject: [Swift-devel] kickstart?
In-Reply-To: <AANLkTi=zcZC9OP92zieJmvhy4EGjNs9z_b_pjenQ-sBc@mail.gmail.com>
Message-ID: <1194909844.857721286494026851.JavaMail.root@zimbra.anl.gov>

Ive not used it, nor have I suggested it to users, in years. 

I'm in favor of deprecating or removing it. Part of the message/diagnostic improvement process could include enhancing the "info" logs and better documenting how they work and how to use them. 

It would be good, I feel, if info logs had as documented keyword/value format as an option that makes it easier to post-process them as a data stream from one or more swift runs. 

- Mike 


----- "Sarah Kenny" <skenny at uchicago.edu> wrote: 
> anyone currently using kickstart? 
> 
> i ask bcs it's in the doc (and mentioned in one of the bugs) but my group has never used it on teragrid (not since the days of vds :) also i don't see references to it on the user list so i was curious if it's still in use? 
> 
> ~sk 
> 
> _______________________________________________ Swift-devel mailing list Swift-devel at ci.uchicago.edu http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel 

-- 
Michael Wilde 
Computation Institute, University of Chicago 
Mathematics and Computer Science Division 
Argonne National Laboratory 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101007/a2f3e50f/attachment.html>

From bugzilla-daemon at mcs.anl.gov  Thu Oct  7 18:47:23 2010
From: bugzilla-daemon at mcs.anl.gov (bugzilla-daemon at mcs.anl.gov)
Date: Thu,  7 Oct 2010 18:47:23 -0500 (CDT)
Subject: [Swift-devel] [Bug 111] stage out -info and cluster logs in the
 same fashion as kickstart records.
In-Reply-To: <bug-111-21@http.bugzilla.mcs.anl.gov/swift/>
References: <bug-111-21@http.bugzilla.mcs.anl.gov/swift/>
Message-ID: <20101007234723.0E7EB2CA25@wind.mcs.anl.gov>

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=111


skenny <skenny at uchicago.edu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |skenny at uchicago.edu
         Resolution|                            |WONTFIX




--- Comment #2 from skenny <skenny at uchicago.edu>  2010-10-07 18:47:22 ---
this is done for info logs and clustering and kickstart are being deprecated

-- 
Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.
You are watching the assignee of the bug.
You are watching the reporter.


From bugzilla-daemon at mcs.anl.gov  Thu Oct  7 19:01:39 2010
From: bugzilla-daemon at mcs.anl.gov (bugzilla-daemon at mcs.anl.gov)
Date: Thu,  7 Oct 2010 19:01:39 -0500 (CDT)
Subject: [Swift-devel] [Bug 112] error reporting in procedure declarations
In-Reply-To: <bug-112-21@http.bugzilla.mcs.anl.gov/swift/>
References: <bug-112-21@http.bugzilla.mcs.anl.gov/swift/>
Message-ID: <20101008000139.C836D2CA2B@wind.mcs.anl.gov>

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=112


skenny <skenny at uchicago.edu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |skenny at uchicago.edu
         Resolution|                            |FIXED




--- Comment #1 from skenny <skenny at uchicago.edu>  2010-10-07 19:01:39 ---
this currently seems like a reasonable error message:

[skenny at communicado]$ swift t1.swift
Could not compile SwiftScript source: line 4:18: expecting an identifier, found
'stdout'

-- 
Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.
You are watching the assignee of the bug.
You are watching the reporter.


From bugzilla-daemon at mcs.anl.gov  Thu Oct  7 19:06:22 2010
From: bugzilla-daemon at mcs.anl.gov (bugzilla-daemon at mcs.anl.gov)
Date: Thu,  7 Oct 2010 19:06:22 -0500 (CDT)
Subject: [Swift-devel] [Bug 120] kickstart transfer errors mask underlying
 application errors
In-Reply-To: <bug-120-21@http.bugzilla.mcs.anl.gov/swift/>
References: <bug-120-21@http.bugzilla.mcs.anl.gov/swift/>
Message-ID: <20101008000622.79C1E2CA2B@wind.mcs.anl.gov>

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=120


skenny <skenny at uchicago.edu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |skenny at uchicago.edu
         Resolution|                            |WONTFIX




--- Comment #1 from skenny <skenny at uchicago.edu>  2010-10-07 19:06:22 ---
kickstart is being deprecated.

-- 
Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.
You are watching the assignee of the bug.
You are watching the reporter.


From wilde at mcs.anl.gov  Thu Oct  7 21:31:45 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 7 Oct 2010 20:31:45 -0600 (GMT-06:00)
Subject: [Swift-devel] Coaster worker connect problem and control over
	coaster logging?
In-Reply-To: <2068606027.857261286492102885.JavaMail.root@zimbra.anl.gov>
Message-ID: <294479349.859881286505105400.JavaMail.root@zimbra.anl.gov>

Found the worker connect problem: the order of interfaces passed was such that the first failed, and the reconnect logic did not clear the failure flag correctly.  Fixed in cog 2903.

The logging problem still needs to be investigated.

- Mike


----- wilde at mcs.anl.gov wrote:

> Im debugging a problem I just started seeing in coasters on PADS. Im
> getting "error code 29" returned from a simple 1-cat job.
> 
> Whats happening is that the coaster worker is failing to connect. Its
> possible I broke it in a recent commit; I hope to know soon. I see
> this in the worker log:
> 
> 1286490899.535 DEBUG - Trying 169.254.95.119:35151...
> 1286490899.571 DEBUG - Connection failed: Connection refused. Trying
> other addresses
> 1286490899.571 DEBUG - Trying 172.5.86.5:35151...
> 1286490899.573 DEBUG - Connection failed: Illegal seek. Trying other
> addresses
> 1286490899.573 DEBUG - Trying 192.5.86.5:35151...
> 1286490899.574 DEBUG - Connection failed: Illegal seek. Trying other
> addresses
> 1286490899.574 ERROR - Connection failed for all addresses. Retrying
> in 1 seconds
> 
> A second question here is about control of the worker log. I see the
> env variable WORKER_LOGGING_ENABLED getting set in the coaster pbs
> submit file. But as far as I can tell, this will not be picked up by
> the worker unless its exported.
> 
> Has this always been set this way? Is anyone actually *getting* worker
> logs in their ~/.globus/coasters directory using trunk?
> 
> I will look further into this; in the meantime Im forcing TRACE
> logging on in worker.pl (which is how I finally got the messages
> above).
> 
> Sarah: this is an interesting and challenging case in error reporting.
> Diagnosing this involves tracking the error from swift stdout to the
> pbs stderr file (with debug=true in the etc/provider-pbs.properties
> file) to the coaster worker log (with elevated logging levels).  Lets
> discuss how the defaults in all these 3 places could be better, and
> how the relevant files could be better coalesced for the user, and
> perhaps integrated by some post-processing diagnostic tool.
> 
> - Mike
> 
> login1$ cat *54.submit
> #PBS -S /bin/sh
> #PBS -N Block-1007-340807-000000
> #PBS -m n
> #PBS -l nodes=1
> #PBS -l walltime=01:00:00
> #PBS -q short
> #PBS -o
> /home/wilde/.globus/scripts/PBS3642641579913160354.submit.stdout
> #PBS -e
> /home/wilde/.globus/scripts/PBS3642641579913160354.submit.stderr
> WORKER_LOGGING_ENABLED=true
> cd / && /usr/bin/perl
> /home/wilde/.globus/coasters/cscript4619936640935778716.pl
> http://169.254.95.119:37300,http://172.5.86.5:37300,http://192.5.86.5:37300
> 1007-340807-000000 /home/wilde/.globus/coasters
> /bin/echo $?
> >/home/wilde/.globus/scripts/PBS3642641579913160354.submit.exitcode
> login1$ 
> 
> 
> -- 
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From bugzilla-daemon at mcs.anl.gov  Fri Oct  8 08:49:58 2010
From: bugzilla-daemon at mcs.anl.gov (bugzilla-daemon at mcs.anl.gov)
Date: Fri,  8 Oct 2010 08:49:58 -0500 (CDT)
Subject: [Swift-devel] [Bug 226] New: Control coaster worker logging via
	swift property
Message-ID: <bug-226-21@http.bugzilla.mcs.anl.gov/swift/>

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=226

           Summary: Control coaster worker logging via swift property
           Product: Swift
           Version: unspecified
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: normal
          Priority: P2
         Component: SwiftScript language
        AssignedTo: skenny at uchicago.edu
        ReportedBy: wilde at mcs.anl.gov


The coaster worker log mechanism needs the following improvements:

The current coaster worker.pl log level is set manually in the source code.

Worker logging is enabled when a coaster block has 16 nodes or more, disabled
otherwise.

Worker logging was - seemingly incorrectly - enabled by setting an environment
variable which doesnt seem to reach the worker.pl script because it is not
exported in the PBS submit file (at least in the case of the PBS provider, as
in jobmanager=local:pbs. Perhaps that works in other provider combinations.

There is an undocumented "force" property to set worker logging on; its not
clear to me if this is settable from swift.properties or not.

I propose that this change to:

1) eliminate the check for 16 workers or more

2) control worker logging solely with a property set in a Karajan namespace in
each coaster-based sites.xml entry.

3) allow logging to be set to a specific level from INFO through TRACE (or
whatever the full range is in the code)

karajan::workerlogging={off,info,debug,trace,etc etc

-- 
Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the reporter.


From wilde at mcs.anl.gov  Fri Oct  8 08:53:09 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Fri, 8 Oct 2010 07:53:09 -0600 (GMT-06:00)
Subject: [Swift-devel] Coaster worker connect problem and control over
	coaster logging?
In-Reply-To: <294479349.859881286505105400.JavaMail.root@zimbra.anl.gov>
Message-ID: <1740032947.868951286545989158.JavaMail.root@zimbra.anl.gov>

Ive filed the coaster logging issue as bug 226.

Sarah, I think that this would make a very good first code change to work on. Can you take a look at it an start a bugzilla thread to discuss?

Thanks,

- Mike



----- "Michael Wilde" <wilde at mcs.anl.gov> wrote:

> Found the worker connect problem: the order of interfaces passed was
> such that the first failed, and the reconnect logic did not clear the
> failure flag correctly.  Fixed in cog 2903.
> 
> The logging problem still needs to be investigated.
> 
> - Mike
> 
> 
> ----- wilde at mcs.anl.gov wrote:
> 
> > Im debugging a problem I just started seeing in coasters on PADS. Im
> > getting "error code 29" returned from a simple 1-cat job.
> >
> > Whats happening is that the coaster worker is failing to connect.
> Its
> > possible I broke it in a recent commit; I hope to know soon. I see
> > this in the worker log:
> >
> > 1286490899.535 DEBUG - Trying 169.254.95.119:35151...
> > 1286490899.571 DEBUG - Connection failed: Connection refused. Trying
> > other addresses
> > 1286490899.571 DEBUG - Trying 172.5.86.5:35151...
> > 1286490899.573 DEBUG - Connection failed: Illegal seek. Trying other
> > addresses
> > 1286490899.573 DEBUG - Trying 192.5.86.5:35151...
> > 1286490899.574 DEBUG - Connection failed: Illegal seek. Trying other
> > addresses
> > 1286490899.574 ERROR - Connection failed for all addresses. Retrying
> > in 1 seconds
> >
> > A second question here is about control of the worker log. I see the
> > env variable WORKER_LOGGING_ENABLED getting set in the coaster pbs
> > submit file. But as far as I can tell, this will not be picked up by
> > the worker unless its exported.
> >
> > Has this always been set this way? Is anyone actually *getting*
> worker
> > logs in their ~/.globus/coasters directory using trunk?
> >
> > I will look further into this; in the meantime Im forcing TRACE
> > logging on in worker.pl (which is how I finally got the messages
> > above).
> >
> > Sarah: this is an interesting and challenging case in error
> reporting.
> > Diagnosing this involves tracking the error from swift stdout to the
> > pbs stderr file (with debug=true in the etc/provider-pbs.properties
> > file) to the coaster worker log (with elevated logging levels). 
> Lets
> > discuss how the defaults in all these 3 places could be better, and
> > how the relevant files could be better coalesced for the user, and
> > perhaps integrated by some post-processing diagnostic tool.
> >
> > - Mike
> >
> > login1$ cat *54.submit
> > #PBS -S /bin/sh
> > #PBS -N Block-1007-340807-000000
> > #PBS -m n
> > #PBS -l nodes=1
> > #PBS -l walltime=01:00:00
> > #PBS -q short
> > #PBS -o
> > /home/wilde/.globus/scripts/PBS3642641579913160354.submit.stdout
> > #PBS -e
> > /home/wilde/.globus/scripts/PBS3642641579913160354.submit.stderr
> > WORKER_LOGGING_ENABLED=true
> > cd / && /usr/bin/perl
> > /home/wilde/.globus/coasters/cscript4619936640935778716.pl
> >
> http://169.254.95.119:37300,http://172.5.86.5:37300,http://192.5.86.5:37300
> > 1007-340807-000000 /home/wilde/.globus/coasters
> > /bin/echo $?
> > >/home/wilde/.globus/scripts/PBS3642641579913160354.submit.exitcode
> > login1$
> >
> >
> > --
> > Michael Wilde
> > Computation Institute, University of Chicago
> > Mathematics and Computer Science Division
> > Argonne National Laboratory
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> 
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From bugzilla-daemon at mcs.anl.gov  Fri Oct  8 10:27:17 2010
From: bugzilla-daemon at mcs.anl.gov (bugzilla-daemon at mcs.anl.gov)
Date: Fri,  8 Oct 2010 10:27:17 -0500 (CDT)
Subject: [Swift-devel] [Bug 227] New: Always keep submit and stdout/err
 files for failing jobs from localscheduler provider
Message-ID: <bug-227-21@http.bugzilla.mcs.anl.gov/swift/>

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=227

           Summary: Always keep submit and stdout/err files for failing
                    jobs from localscheduler provider
           Product: Swift
           Version: unspecified
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: normal
          Priority: P2
         Component: SwiftScript language
        AssignedTo: skenny at uchicago.edu
        ReportedBy: wilde at mcs.anl.gov


This should happen even without the debug=true flag in e.g.
etc/providers-pbs.properties.

Its possible that what Im seeing (ie that these files are not being retained)
is due to the fact the its a coaster worker job launch that Im current;y
debugging. But currently I need to keep setting debug-true to get these files
retained when the coaster worker job launch fails.

Its possible that the desired file retention is happening correctly for
ordinary PBS jobs.

Same applies to Condor and SGE local scheduler providers, and ideally to Globus
provider jobs.

Also the debug=true flag should be documented in a User Guide "Troubleshooting"
section.

-- 
Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the reporter.


From dk0966 at cs.ship.edu  Fri Oct  8 14:00:02 2010
From: dk0966 at cs.ship.edu (dk0966 at cs.ship.edu)
Date: Fri, 8 Oct 2010 14:00:02 -0500
Subject: [Swift-devel] Swift usage summary
Message-ID: <20101008190002.A34F78D00072@bridled.ci.uchicago.edu>

An embedded and charset-unspecified text was scrubbed...
Name: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101008/6f50b930/attachment.ksh>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: usage.png
Type: image/png
Size: 9590 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101008/6f50b930/attachment.png>

From wilde at mcs.anl.gov  Fri Oct  8 14:22:13 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Fri, 8 Oct 2010 13:22:13 -0600 (GMT-06:00)
Subject: [Swift-devel] Re: Swift usage summary
In-Reply-To: <20101008190002.A34F78D00072@bridled.ci.uchicago.edu>
Message-ID: <683670089.889701286565733914.JavaMail.root@zimbra.anl.gov>

Very nice, David.

Is it easy to add a few more summary stats on top, like #runs, #apps run, avg run duration, %runs that made it to swift end, % runs that finished w/o errors, etc?

Also, how to validate the date, especially the 51 unique users?

How to illustrate where the hosts are, by geography and domain?

I would only do whats easy in this regard; at some point soon we need to assess if/when to integrate the Globus tools that will do some of this for us.

Thanks,

Mike


----- dk0966 at cs.ship.edu wrote:

> Number of unique hosts in the last 30 days: 35
> 
> Number of unique users in the last 30 days: 51
> 
> 
> 
> [image/png:usage.png]

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From aespinosa at cs.uchicago.edu  Fri Oct  8 16:48:27 2010
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Fri, 8 Oct 2010 16:48:27 -0500
Subject: [Swift-devel] Patch: Local service port option (was Re: [Swift-user]
 coaster-service local service port)
In-Reply-To: <20101008214157.GD2802@origin>
References: <20101008211026.GC2802@origin>
	<1637738404.897251286573439520.JavaMail.root@zimbra.anl.gov>
	<20101008214157.GD2802@origin>
Message-ID: <20101008214827.GE2802@origin>

Descrition: Adds "-localport <localport>" option to bin/coaster-service.

Patch found in http://www.ci.uchicago.edu/~aespinosa/patches/cogkit-coaster_local_port.patch

Enjoy!
-Allan


On Fri, Oct 08, 2010 at 04:41:57PM -0500, Allan Espinosa wrote:
> Thanks Mike.
> 
> Poking at CoasterPersistentService.java, it looks like the port 50000+ for the
> local service is automatically set by the provider.  I'm coding up a patch to
> add the option.  
> 
> Hopefully this won't break things :)
> 
> -Allan
> 
> On Fri, Oct 08, 2010 at 03:30:39PM -0600, Michael Wilde wrote:
> > Allan, the only help I know of for the command is:
> > 
> > com$ coaster-service -help
> > Usage:
> >   coaster-service <options> 
> > 
> >         where options are:
> > 
> >     [(-port | -p) <port>]
> >       Specifies which port to start the service on 
> > 
> >     [-nosec]
> >       Disables GSI security and uses plain TCP sockets instead 
> > 
> >     [-proxy <file>]
> >       Specifies the location of a proxy credential that will be used 
> >       for authentication. If not specified, the default proxy will be 
> >       used. 
> > 
> >     [-local]
> >       Binds the service to the loopback interface 
> > 
> >     [(-help | -h)]
> >       Displays usage information 
> > 
> > 
> > Mike
> 
-- 
Allan M. Espinosa <http://amespinosa.wordpress.com>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>


From hategan at mcs.anl.gov  Fri Oct  8 23:11:51 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 08 Oct 2010 21:11:51 -0700
Subject: [Swift-devel] Patch: Local service port option (was Re:
	[Swift-user] coaster-service local service port)
In-Reply-To: <20101008214827.GE2802@origin>
References: <20101008211026.GC2802@origin>
	<1637738404.897251286573439520.JavaMail.root@zimbra.anl.gov>
	<20101008214157.GD2802@origin>  <20101008214827.GE2802@origin>
Message-ID: <1286597511.2681.9.camel@blabla2.none>

Let me (and possibly others) get this straight:

- the persistent service has two ports:
  1. the port to which a client connects to
  2. the port to which the workers connect to
- there is a command line option for specifying (1)
- you need a command line option for specifying (2)

Right?

On Fri, 2010-10-08 at 16:48 -0500, Allan Espinosa wrote:
> Descrition: Adds "-localport <localport>" option to bin/coaster-service.
> 
> Patch found in http://www.ci.uchicago.edu/~aespinosa/patches/cogkit-coaster_local_port.patch
> 
> Enjoy!
> -Allan
> 
> 
> On Fri, Oct 08, 2010 at 04:41:57PM -0500, Allan Espinosa wrote:
> > Thanks Mike.
> > 
> > Poking at CoasterPersistentService.java, it looks like the port 50000+ for the
> > local service is automatically set by the provider.  I'm coding up a patch to
> > add the option.  
> > 
> > Hopefully this won't break things :)
> > 
> > -Allan
> > 
> > On Fri, Oct 08, 2010 at 03:30:39PM -0600, Michael Wilde wrote:
> > > Allan, the only help I know of for the command is:
> > > 
> > > com$ coaster-service -help
> > > Usage:
> > >   coaster-service <options> 
> > > 
> > >         where options are:
> > > 
> > >     [(-port | -p) <port>]
> > >       Specifies which port to start the service on 
> > > 
> > >     [-nosec]
> > >       Disables GSI security and uses plain TCP sockets instead 
> > > 
> > >     [-proxy <file>]
> > >       Specifies the location of a proxy credential that will be used 
> > >       for authentication. If not specified, the default proxy will be 
> > >       used. 
> > > 
> > >     [-local]
> > >       Binds the service to the loopback interface 
> > > 
> > >     [(-help | -h)]
> > >       Displays usage information 
> > > 
> > > 
> > > Mike
> > 




From aespinosa at cs.uchicago.edu  Fri Oct  8 23:13:07 2010
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Fri, 8 Oct 2010 23:13:07 -0500
Subject: [Swift-devel] Patch: Local service port option (was Re:
	[Swift-user] coaster-service local service port)
In-Reply-To: <1286597511.2681.9.camel@blabla2.none>
References: <20101008211026.GC2802@origin>
	<1637738404.897251286573439520.JavaMail.root@zimbra.anl.gov>
	<20101008214157.GD2802@origin> <20101008214827.GE2802@origin>
	<1286597511.2681.9.camel@blabla2.none>
Message-ID: <AANLkTin6GKJOmviEEhu3Py7R42YW-TF4HLaVRUamw_mt@mail.gmail.com>

Yes. I do need an option for (2)

2010/10/8 Mihael Hategan <hategan at mcs.anl.gov>:
> Let me (and possibly others) get this straight:
>
> - the persistent service has two ports:
> ?1. the port to which a client connects to
> ?2. the port to which the workers connect to
> - there is a command line option for specifying (1)
> - you need a command line option for specifying (2)
>
> Right?

-- 
Allan M. Espinosa <http://amespinosa.wordpress.com>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>


From hategan at mcs.anl.gov  Fri Oct  8 23:15:05 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 08 Oct 2010 21:15:05 -0700
Subject: [Swift-devel] Patch: Local service port option (was Re:
	[Swift-user] coaster-service local service port)
In-Reply-To: <AANLkTin6GKJOmviEEhu3Py7R42YW-TF4HLaVRUamw_mt@mail.gmail.com>
References: <20101008211026.GC2802@origin>
	<1637738404.897251286573439520.JavaMail.root@zimbra.anl.gov>
	<20101008214157.GD2802@origin> <20101008214827.GE2802@origin>
	<1286597511.2681.9.camel@blabla2.none>
	<AANLkTin6GKJOmviEEhu3Py7R42YW-TF4HLaVRUamw_mt@mail.gmail.com>
Message-ID: <1286597705.2681.10.camel@blabla2.none>

Your patch looks right then. I shall commit this tomorrow (unless
somebody beats me to it).

On Fri, 2010-10-08 at 23:13 -0500, Allan Espinosa wrote:
> Yes. I do need an option for (2)
> 
> 2010/10/8 Mihael Hategan <hategan at mcs.anl.gov>:
> > Let me (and possibly others) get this straight:
> >
> > - the persistent service has two ports:
> >  1. the port to which a client connects to
> >  2. the port to which the workers connect to
> > - there is a command line option for specifying (1)
> > - you need a command line option for specifying (2)
> >
> > Right?
> 




From hategan at mcs.anl.gov  Sat Oct  9 13:38:34 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sat, 09 Oct 2010 11:38:34 -0700
Subject: [Swift-devel] Patch: Local service port option (was Re:
	[Swift-user] coaster-service local service port)
In-Reply-To: <1286597705.2681.10.camel@blabla2.none>
References: <20101008211026.GC2802@origin>
	<1637738404.897251286573439520.JavaMail.root@zimbra.anl.gov>
	<20101008214157.GD2802@origin> <20101008214827.GE2802@origin>
	<1286597705.2681.10.camel@blabla2.none>
Message-ID: <1286649514.6823.0.camel@blabla2.none>

On Fri, 2010-10-08 at 21:15 -0700, Mihael Hategan wrote:
> Your patch looks right then. I shall commit this tomorrow (unless
> somebody beats me to it).

Done in cog r2908 (trunk).

> 
> On Fri, 2010-10-08 at 23:13 -0500, Allan Espinosa wrote:
> > Yes. I do need an option for (2)
> > 
> > 2010/10/8 Mihael Hategan <hategan at mcs.anl.gov>:
> > > Let me (and possibly others) get this straight:
> > >
> > > - the persistent service has two ports:
> > >  1. the port to which a client connects to
> > >  2. the port to which the workers connect to
> > > - there is a command line option for specifying (1)
> > > - you need a command line option for specifying (2)
> > >
> > > Right?
> > 
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel




From bugzilla-daemon at mcs.anl.gov  Sat Oct  9 15:41:04 2010
From: bugzilla-daemon at mcs.anl.gov (bugzilla-daemon at mcs.anl.gov)
Date: Sat,  9 Oct 2010 15:41:04 -0500 (CDT)
Subject: [Swift-devel] [Bug 226] Control coaster worker logging via swift
	property
In-Reply-To: <bug-226-21@http.bugzilla.mcs.anl.gov/swift/>
References: <bug-226-21@http.bugzilla.mcs.anl.gov/swift/>
Message-ID: <20101009204104.BDA362BF3B@wind.mcs.anl.gov>

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=226


Mihael Hategan <hategan at mcs.anl.gov> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |hategan at mcs.anl.gov
         Resolution|                            |FIXED




--- Comment #1 from Mihael Hategan <hategan at mcs.anl.gov>  2010-10-09 15:41:04 ---
I changed the logging control as follows:
- removed the 16 worker check
- control using task attribute/coaster service setting (in Swift this would be
something like: <profile namespace="globus"
key="workerLoggingLevel">DEBUG</profile>).
- "OFF" and "NONE" both mean the same thing (nothing is logged and no log file
is created).

This still relies on environment variables. If the PBS provider has a problem
with passing environment variables, please file a bug report for the PBS
provider.

-- 
Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching someone on the CC list of the bug.
You are watching the reporter.


From hategan at mcs.anl.gov  Sun Oct 10 17:22:12 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 10 Oct 2010 15:22:12 -0700
Subject: [Swift-devel] fast branch
Message-ID: <1286749332.19982.1.camel@blabla2.none>

For those who care, I fixed two bugs in the fast branch which were
triggered when using normal staging instead of provider staging (which
is why I didn't get to see them in my testing).

Mihael



From wilde at mcs.anl.gov  Mon Oct 11 15:12:14 2010
From: wilde at mcs.anl.gov (wilde at mcs.anl.gov)
Date: Mon, 11 Oct 2010 14:12:14 -0600 (GMT-06:00)
Subject: [Swift-devel] Standard Swift coaster behavior doesnt work well for
	sporadic jobs
In-Reply-To: <713025103.964571286827793614.JavaMail.root@zimbra.anl.gov>
Message-ID: <192374997.964831286827934282.JavaMail.root@zimbra.anl.gov>

Mihael, Justin, does the following sound like a likely coaster issue:

When using the standard Swift coaster code (not passive or persistent), if I have a job that runs at the start, and then there is a long delay before the next job, such that the coaster worker times out, then the coaster scheduler doesnt think that there is a valid block into which the job can fit, and Swift just hangs, with the job in submitted state but never getting assigned to a block.

The behavior seems similar to what you see when you try to run a job that doesnt fit into any block that you have defined using the coasters sites.xml parameters: in that case, too, Swift just hangs.

Both of these situations (whether or not they are indeed due to the same algorithmic issue) seem to be problems that we need to address. In the first case (which is my immediate and more important problem) you can see a log for the problem in ~wilde/swift-rserver-hangs, along with the swift script, tc, sites, and properties  file(cf).

Thanks,

Mike

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From bugzilla-daemon at mcs.anl.gov  Mon Oct 11 17:58:07 2010
From: bugzilla-daemon at mcs.anl.gov (bugzilla-daemon at mcs.anl.gov)
Date: Mon, 11 Oct 2010 17:58:07 -0500 (CDT)
Subject: [Swift-devel] [Bug 228] New: PBS provider is not passing env vars
	correctly
Message-ID: <bug-228-21@http.bugzilla.mcs.anl.gov/swift/>

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=228

           Summary: PBS provider is not passing env vars correctly
           Product: Swift
           Version: unspecified
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: normal
          Priority: P2
         Component: SwiftScript language
        AssignedTo: skenny at uchicago.edu
        ReportedBy: wilde at mcs.anl.gov


Env vars - in particular the var WORKER_LOGGING_ENABLED - are not being passing
correctly.

Currently this var is just added as:

WORKER_LOGGING_ENABLED=true

I think it needs either:

1) to be prefixed by "export " so that it is placed in the environment of the
executed job (worker.pl)

2) be passed in a PBS directive, something like:

#PBS -v WORKER_LOGGING_ENABLED=true

This should be tested first with standalone PBS .sub files and qsub.

-- 
Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the reporter.


From jon.monette at gmail.com  Mon Oct 11 19:44:43 2010
From: jon.monette at gmail.com (Jonathan Monette)
Date: Mon, 11 Oct 2010 19:44:43 -0500
Subject: [Swift-devel] Standard Swift coaster behavior doesnt work well
	for sporadic jobs
In-Reply-To: <192374997.964831286827934282.JavaMail.root@zimbra.anl.gov>
References: <192374997.964831286827934282.JavaMail.root@zimbra.anl.gov>
Message-ID: <4CB3AF7B.4020705@gmail.com>

  Wouldn't coasters just re-submit jobs if there are no workers 
available to process them?   My Montage stuff is under the assumption 
that coasters will submit more workers if they all time out.  This maybe 
why my stuff was hanging before.  Not entirely for sure since I am 
working on another problem.

On 10/11/2010 03:12 PM, wilde at mcs.anl.gov wrote:
> Mihael, Justin, does the following sound like a likely coaster issue:
>
> When using the standard Swift coaster code (not passive or persistent), if I have a job that runs at the start, and then there is a long delay before the next job, such that the coaster worker times out, then the coaster scheduler doesnt think that there is a valid block into which the job can fit, and Swift just hangs, with the job in submitted state but never getting assigned to a block.
>
> The behavior seems similar to what you see when you try to run a job that doesnt fit into any block that you have defined using the coasters sites.xml parameters: in that case, too, Swift just hangs.
>
> Both of these situations (whether or not they are indeed due to the same algorithmic issue) seem to be problems that we need to address. In the first case (which is my immediate and more important problem) you can see a log for the problem in ~wilde/swift-rserver-hangs, along with the swift script, tc, sites, and properties  file(cf).
>
> Thanks,
>
> Mike
>

-- 
Jon

Computers are incredibly fast, accurate, and stupid. Human beings are incredibly slow, inaccurate, and brilliant. Together they are powerful beyond imagination.
- Albert Einstein



From wilde at mcs.anl.gov  Mon Oct 11 20:10:36 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Mon, 11 Oct 2010 19:10:36 -0600 (GMT-06:00)
Subject: [Swift-devel] Standard Swift coaster behavior doesnt work well
	for sporadic jobs
In-Reply-To: <4CB3AF7B.4020705@gmail.com>
Message-ID: <57986955.982201286845836821.JavaMail.root@zimbra.anl.gov>


----- "Jonathan Monette" <jon.monette at gmail.com> wrote:

> Wouldn't coasters just re-submit jobs if there are no workers 
> available to process them? 

Thats certainly the desired behavior for the default "automatic" mode, but it doesnt appear to be working that way - unless Ive broken it with a local mod.

- Mike


> My Montage stuff is under the assumption
> 
> that coasters will submit more workers if they all time out.  This
> maybe 
> why my stuff was hanging before.  Not entirely for sure since I am 
> working on another problem.


> On 10/11/2010 03:12 PM, wilde at mcs.anl.gov wrote:
> > Mihael, Justin, does the following sound like a likely coaster
> issue:
> >
> > When using the standard Swift coaster code (not passive or
> persistent), if I have a job that runs at the start, and then there is
> a long delay before the next job, such that the coaster worker times
> out, then the coaster scheduler doesnt think that there is a valid
> block into which the job can fit, and Swift just hangs, with the job
> in submitted state but never getting assigned to a block.
> >
> > The behavior seems similar to what you see when you try to run a job
> that doesnt fit into any block that you have defined using the
> coasters sites.xml parameters: in that case, too, Swift just hangs.
> >
> > Both of these situations (whether or not they are indeed due to the
> same algorithmic issue) seem to be problems that we need to address.
> In the first case (which is my immediate and more important problem)
> you can see a log for the problem in ~wilde/swift-rserver-hangs, along
> with the swift script, tc, sites, and properties  file(cf).
> >
> > Thanks,
> >
> > Mike
> >
> 
> -- 
> Jon
> 
> Computers are incredibly fast, accurate, and stupid. Human beings are
> incredibly slow, inaccurate, and brilliant. Together they are powerful
> beyond imagination.
> - Albert Einstein

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From bugzilla-daemon at mcs.anl.gov  Mon Oct 11 21:03:47 2010
From: bugzilla-daemon at mcs.anl.gov (bugzilla-daemon at mcs.anl.gov)
Date: Mon, 11 Oct 2010 21:03:47 -0500 (CDT)
Subject: [Swift-devel] [Bug 228] PBS provider is not passing env vars
	correctly
In-Reply-To: <bug-228-21@http.bugzilla.mcs.anl.gov/swift/>
References: <bug-228-21@http.bugzilla.mcs.anl.gov/swift/>
Message-ID: <20101012020347.12C022CB20@wind.mcs.anl.gov>

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=228


Mihael Hategan <hategan at mcs.anl.gov> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |hategan at mcs.anl.gov
         Resolution|                            |FIXED




--- Comment #1 from Mihael Hategan <hategan at mcs.anl.gov>  2010-10-11 21:03:46 ---
Fixed in trunk/cog r2913 by adding all environment variable names to #PBS -v as
documented in http://doesciencegrid.org/public/pbs/qsub.html

-- 
Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching someone on the CC list of the bug.
You are watching the reporter.


From wilde at mcs.anl.gov  Tue Oct 12 10:14:56 2010
From: wilde at mcs.anl.gov (wilde at mcs.anl.gov)
Date: Tue, 12 Oct 2010 09:14:56 -0600 (GMT-06:00)
Subject: [Swift-devel] Standard Swift coaster behavior doesnt work well
	for sporadic jobs
In-Reply-To: <883891837.966851286829632605.JavaMail.root@zimbra.anl.gov>
Message-ID: <1496081715.1007261286896496622.JavaMail.root@zimbra.anl.gov>

I see:

2010-10-11 14:54:35,010-0500 INFO ?CoasterService Started coaster service: http://192.5.86.5:34445
2010-10-11 14:54:35,021-0500 INFO ?Command Sending Command(2, SUBMITJOB) on null[244757954: {}]
2010-10-11 14:54:35,080-0500 INFO ?BlockQueueProcessor allocsize = 0.0, queuedsize = 1.0660596665516473, qsz = 1
2010-10-11 14:54:35,081-0500 INFO ?BlockQueueProcessor Requeued 1 non-fitting jobs
2010-10-11 14:54:35,082-0500 INFO ?BlockQueueProcessor
Settings {
?? ? ? ?slots = 32
?? ? ? ?workersPerNode = 1
?? 

but for the second job, I see:

2010-10-11 14:59:55,200-0500 INFO ?Command Sending Command(3, SUBMITJOB) on null[244757954: {}]
2010-10-11 14:59:55,224-0500 INFO ?Cpu 1011-540235-000000:0 pull
2010-10-11 14:59:55,225-0500 INFO ?Cpu 1011-540235-000000:0 submitting urn:1286826874044-1286826874046-1286826874047
2010-10-11 14:59:55,226-0500 INFO ?Command Sending Command(3, SUBMITJOB) on SC-1011-540235-000000-000000
2010-10-11 14:59:55,226-0500 INFO ?AbstractStreamKarajanChannel Sender 390276053 queue size: 0
2010-10-11 14:59:56,620-0500 INFO ?BlockQueueProcessor allocsize = 0.0, queuedsize = 0.0, qsz = 0
2010-10-11 14:59:56,620-0500 INFO ?BlockQueueProcessor Plan time: 0
2010-10-11 14:59:58,822-0500 INFO ?BlockQueueProcessor allocsize = 0.0, queuedsize = 0.0, qsz = 0
2010-10-11 14:59:58,822-0500 INFO ?BlockQueueProcessor Plan time: 0
2010-10-11 14:59:59,935-0500 INFO ?AbstractStreamKarajanChannel$Multiplexer Avg stream buf: 0
2010-10-11 15:00:00,790-0500 INFO ?Cpu runTime: 2, sleepTime: 10036
2010-10-11 15:00:01,024-0500 INFO ?BlockQueueProcessor allocsize = 0.0, queuedsize = 0.0, qsz = 0
2010-10-11 15:00:01,024-0500 INFO ?BlockQueueProcessor Plan time: 0
2010-10-11 15:00:03,226-0500 INFO ?BlockQueueProcessor allocsize = 0.0, queuedsize = 0.0, qsz = 0
2010-10-11 15:00:03,226-0500 INFO ?BlockQueueProcessor Plan time: 0
2010-10-11 15:00:05,112-0500 INFO ?CoasterService Idle time: 0
2010-10-11 15:00:05,122-0500 INFO ?TaskNotifier Congestion queue size: 0


...which suggests that the coaster service doesnt really see the job in the queue?

The one mod I made that may be causing this was to set the service-side timeout value for the coaster provider up high; this was needed to keep manually configured passive-persistent configurations alive while the Swift client was idle (eg for the multiple-ssh-server configuration of the R server).

- Mike


----- wilde at mcs.anl.gov wrote:

> Mihael, Justin, does the following sound like a likely coaster issue:
> 
> When using the standard Swift coaster code (not passive or
> persistent), if I have a job that runs at the start, and then there is
> a long delay before the next job, such that the coaster worker times
> out, then the coaster scheduler doesnt think that there is a valid
> block into which the job can fit, and Swift just hangs, with the job
> in submitted state but never getting assigned to a block.
> 
> The behavior seems similar to what you see when you try to run a job
> that doesnt fit into any block that you have defined using the
> coasters sites.xml parameters: in that case, too, Swift just hangs.
> 
> Both of these situations (whether or not they are indeed due to the
> same algorithmic issue) seem to be problems that we need to address.
> In the first case (which is my immediate and more important problem)
> you can see a log for the problem in ~wilde/swift-rserver-hangs, along
> with the swift script, tc, sites, and properties ?file(cf).
> 
> Thanks,
> 
> Mike
> 
> -- 
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From wilde at mcs.anl.gov  Tue Oct 12 10:26:25 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Tue, 12 Oct 2010 09:26:25 -0600 (GMT-06:00)
Subject: [Swift-devel] Standard Swift coaster behavior doesnt work well
	for sporadic jobs
In-Reply-To: <1496081715.1007261286896496622.JavaMail.root@zimbra.anl.gov>
Message-ID: <1033022032.1008581286897185877.JavaMail.root@zimbra.anl.gov>

I forgot to say: I will retest on an unmodified trunk to see if this problem is innate in the latest code or introduced by my mod.

- Mike

----- wilde at mcs.anl.gov wrote:

> I see:
> 
> 2010-10-11 14:54:35,010-0500 INFO ?CoasterService Started coaster
> service: http://192.5.86.5:34445
> 2010-10-11 14:54:35,021-0500 INFO ?Command Sending Command(2,
> SUBMITJOB) on null[244757954: {}]
> 2010-10-11 14:54:35,080-0500 INFO ?BlockQueueProcessor allocsize =
> 0.0, queuedsize = 1.0660596665516473, qsz = 1
> 2010-10-11 14:54:35,081-0500 INFO ?BlockQueueProcessor Requeued 1
> non-fitting jobs
> 2010-10-11 14:54:35,082-0500 INFO ?BlockQueueProcessor
> Settings {
> ?? ? ? ?slots = 32
> ?? ? ? ?workersPerNode = 1
> ?? 
> 
> but for the second job, I see:
> 
> 2010-10-11 14:59:55,200-0500 INFO ?Command Sending Command(3,
> SUBMITJOB) on null[244757954: {}]
> 2010-10-11 14:59:55,224-0500 INFO ?Cpu 1011-540235-000000:0 pull
> 2010-10-11 14:59:55,225-0500 INFO ?Cpu 1011-540235-000000:0 submitting
> urn:1286826874044-1286826874046-1286826874047
> 2010-10-11 14:59:55,226-0500 INFO ?Command Sending Command(3,
> SUBMITJOB) on SC-1011-540235-000000-000000
> 2010-10-11 14:59:55,226-0500 INFO ?AbstractStreamKarajanChannel Sender
> 390276053 queue size: 0
> 2010-10-11 14:59:56,620-0500 INFO ?BlockQueueProcessor allocsize =
> 0.0, queuedsize = 0.0, qsz = 0
> 2010-10-11 14:59:56,620-0500 INFO ?BlockQueueProcessor Plan time: 0
> 2010-10-11 14:59:58,822-0500 INFO ?BlockQueueProcessor allocsize =
> 0.0, queuedsize = 0.0, qsz = 0
> 2010-10-11 14:59:58,822-0500 INFO ?BlockQueueProcessor Plan time: 0
> 2010-10-11 14:59:59,935-0500 INFO
> ?AbstractStreamKarajanChannel$Multiplexer Avg stream buf: 0
> 2010-10-11 15:00:00,790-0500 INFO ?Cpu runTime: 2, sleepTime: 10036
> 2010-10-11 15:00:01,024-0500 INFO ?BlockQueueProcessor allocsize =
> 0.0, queuedsize = 0.0, qsz = 0
> 2010-10-11 15:00:01,024-0500 INFO ?BlockQueueProcessor Plan time: 0
> 2010-10-11 15:00:03,226-0500 INFO ?BlockQueueProcessor allocsize =
> 0.0, queuedsize = 0.0, qsz = 0
> 2010-10-11 15:00:03,226-0500 INFO ?BlockQueueProcessor Plan time: 0
> 2010-10-11 15:00:05,112-0500 INFO ?CoasterService Idle time: 0
> 2010-10-11 15:00:05,122-0500 INFO ?TaskNotifier Congestion queue size:
> 0
> 
> 
> ...which suggests that the coaster service doesnt really see the job
> in the queue?
> 
> The one mod I made that may be causing this was to set the
> service-side timeout value for the coaster provider up high; this was
> needed to keep manually configured passive-persistent configurations
> alive while the Swift client was idle (eg for the multiple-ssh-server
> configuration of the R server).
> 
> - Mike
> 
> 
> ----- wilde at mcs.anl.gov wrote:
> 
> > Mihael, Justin, does the following sound like a likely coaster
> issue:
> > 
> > When using the standard Swift coaster code (not passive or
> > persistent), if I have a job that runs at the start, and then there
> is
> > a long delay before the next job, such that the coaster worker
> times
> > out, then the coaster scheduler doesnt think that there is a valid
> > block into which the job can fit, and Swift just hangs, with the
> job
> > in submitted state but never getting assigned to a block.
> > 
> > The behavior seems similar to what you see when you try to run a
> job
> > that doesnt fit into any block that you have defined using the
> > coasters sites.xml parameters: in that case, too, Swift just hangs.
> > 
> > Both of these situations (whether or not they are indeed due to the
> > same algorithmic issue) seem to be problems that we need to
> address.
> > In the first case (which is my immediate and more important
> problem)
> > you can see a log for the problem in ~wilde/swift-rserver-hangs,
> along
> > with the swift script, tc, sites, and properties ?file(cf).
> > 
> > Thanks,
> > 
> > Mike
> > 
> > -- 
> > Michael Wilde
> > Computation Institute, University of Chicago
> > Mathematics and Computer Science Division
> > Argonne National Laboratory
> > 
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> 
> -- 
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From aespinosa at cs.uchicago.edu  Tue Oct 12 13:54:42 2010
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Tue, 12 Oct 2010 13:54:42 -0500
Subject: [Swift-devel] missing perl libraries on OSG sites.
Message-ID: <20101012185442.GA2510@origin>

Hi,

Here are some sites that has problems running the perl workers because of
missing perl libraries:

UCHC_CBG
SMU_PHY

The error message is,
010-10-12 13:40:26,068-0500 INFO  vdl:execute Exception in worker5:
Arguments: [http://128.135.125.17:61005, PRELIM, /tmp]
Host: UCHC_CBG
Directory: worker-trabaho/jobs/c/worker5-cmqdb20k
stderr.txt: Can't locate Time/HiRes.pm in @INC (@INC contains:
/usr/lib/perl5/5.8.5/i386-linux-thread-multi /usr/lib/perl5/5.8.5
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi
/usr/lib/perl5/site_perl/5.8.5 /usr/lib/perl5/site_perl
/usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi
/usr/lib/perl5/vendor_perl/5.8.5 /usr/lib/perl5/vendor_perl .) at
/osg/app/engage-scec/worker.pl line 17.
BEGIN failed--compilation aborted at /osg/app/engage-scec/worker.pl line 17.

stdout.txt: 
----

Caused by: Exit code 2

-Allan

-- 
Allan M. Espinosa <http://amespinosa.wordpress.com>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>


From aespinosa at cs.uchicago.edu  Tue Oct 12 16:04:58 2010
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Tue, 12 Oct 2010 16:04:58 -0500
Subject: [Swift-devel] worker.pl IDLETIMEOUT
Message-ID: <20101012210458.GE2510@origin>

Poking at worker.pl, I see that it accepts a third argument for idle time.  Is
this in seconds?

Also, I'm using swift to driver a number of passive workers.  The worker jobs
fail due to this timeout.  I may have to modify things to suit this kind of
setup.

Thanks,
-Allan

-- 
Allan M. Espinosa <http://amespinosa.wordpress.com>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>


From wilde at mcs.anl.gov  Tue Oct 12 18:56:10 2010
From: wilde at mcs.anl.gov (wilde at mcs.anl.gov)
Date: Tue, 12 Oct 2010 17:56:10 -0600 (GMT-06:00)
Subject: [Swift-devel] worker.pl IDLETIMEOUT
In-Reply-To: <1244384433.1050651286927624302.JavaMail.root@zimbra.anl.gov>
Message-ID: <495028650.1050711286927770885.JavaMail.root@zimbra.anl.gov>

I think Ive already got these mods for the worker in trunk.

IDLETIMEOUT is in seconds

I think you need to set the timer up high in the service as well (see diff below).

But Im not sure if this mod is breaking normal coaster behavior (in the normal automated mode) - see my note on that on the list a few posts back.

- Mike


Index: src/org/globus/cog/abstraction/coaster/service/CoasterService.java
===================================================================
--- src/org/globus/cog/abstraction/coaster/service/CoasterService.java  (revision 2902)
+++ src/org/globus/cog/abstraction/coaster/service/CoasterService.java  (working copy)
@@ -41,7 +41,7 @@
     public static final Logger logger = Logger
             .getLogger(CoasterService.class);
 
-    public static final int IDLE_TIMEOUT = 120 * 1000;
+    public static final int IDLE_TIMEOUT = 120 * 1000 /* extend it: */ * 30 * 240;
 
     public static final RequestManager COASTER_REQUEST_MANAGER = new CoasterRequestManager();
 



----- "Allan Espinosa" <aespinosa at cs.uchicago.edu> wrote:

> Poking at worker.pl, I see that it accepts a third argument for idle
> time.  Is
> this in seconds?
> 
> Also, I'm using swift to driver a number of passive workers.  The
> worker jobs
> fail due to this timeout.  I may have to modify things to suit this
> kind of
> setup.
> 
> Thanks,
> -Allan
> 
> -- 
> Allan M. Espinosa <http://amespinosa.wordpress.com>
> PhD student, Computer Science
> University of Chicago <http://people.cs.uchicago.edu/~aespinosa>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From jon.monette at gmail.com  Wed Oct 13 21:12:14 2010
From: jon.monette at gmail.com (Jonathan Monette)
Date: Wed, 13 Oct 2010 21:12:14 -0500
Subject: [Swift-devel] Switch statement
Message-ID: <4CB666FE.9050804@gmail.com>

  Hello,
     If one would like to modify the switch control construct, where in 
the source should he start looking?

-- 
Jon

Computers are incredibly fast, accurate, and stupid. Human beings are incredibly slow, inaccurate, and brilliant. Together they are powerful beyond imagination.
- Albert Einstein



From skenny at uchicago.edu  Thu Oct 14 12:28:10 2010
From: skenny at uchicago.edu (Sarah Kenny)
Date: Thu, 14 Oct 2010 12:28:10 -0500
Subject: [Swift-devel] swift versions
In-Reply-To: <1286489122.28173.0.camel@blabla2.none>
References: <383110182.852171286486770000.JavaMail.root@zimbra.anl.gov>
	<1286489122.28173.0.camel@blabla2.none>
Message-ID: <AANLkTimmLsVFN9_0qEh2bahQ5A+6uHJxrhaKQ5oQGYTj@mail.gmail.com>

so for new changes to the doc (which will eventually go into release 1.1) i
will add them to trunk. am i understanding this correctly in that i should
add a copy of the current doc dir (/ci/www/projects/swift/guides) to
trunk/docs (https://trac.ci.uchicago.edu/swift/browser/trunk/docs) & then
commit my own changes to that?

~sk

On Thu, Oct 7, 2010 at 5:05 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> On Thu, 2010-10-07 at 15:26 -0600, Michael Wilde wrote:
> > ----- "Justin M Wozniak" <wozniak at mcs.anl.gov> wrote:
> >
> > > Yes, but I think that 1.0 is a release candidate- right?  New material
> > >
> > > will go into trunk until we branch for 1.1 .
> >
> > Justin, can you clarify this?  I think we want to take most or all of
> > whats in the trunk *now*, and branch that for 1.0.  That has not been
> > done yet, has it?
>
> It has. For a long time. We have not yet released branches/1.0.
>
> >
> > If not, lets discuss when to make the branch, and especially how to test.
> >
> > If its already been done, lets make sure we have all we want/need from
> trunk in the release branch.
> >
> > I'm eager to see a 1.0 very soon.
> >
> > - Mike
> >
> > >
> > > On Thu, 7 Oct 2010, Sarah Kenny wrote:
> > >
> > > > sounds good.
> > > >
> > > > just want to verify (sorry to beat this to death):
> > > >
> > > > so, we alias the 'main guide' to 0.9 (as suggested) and any changes
> > > we make
> > > > will go to the doc in branches/1.0 which will then be the new alias
> > > for the
> > > > 'main guide' once we do the release.
> > > >
> > > >
> > > >
> > > > On Thu, Oct 7, 2010 at 3:45 PM, Justin M Wozniak
> > > <wozniak at mcs.anl.gov>wrote:
> > > >
> > > >>
> > > >> Yup.
> > > >>
> > > >>
> > > >> On Thu, 7 Oct 2010, Mihael Hategan wrote:
> > > >>
> > > >>  To summarize:
> > > >>>
> > > >>> There is a 1-1 mapping between branches and docs. One of the
> > > branches
> > > >>> (corresponding to the current release) gets linked from "main"
> > > (i.e.
> > > >>> main docs are the docs for the current release). So:
> > > >>>
> > > >>> branches/0.9 <-> 0.9 docs
> > > >>> banches/0.8 <-> 0.8 docs
> > > >>> trunk <-> trunk docs
> > > >>>
> > > >>> If current release is 0.9, then main docs = 0.9 docs.
> > > >>>
> > > >>> On Thu, 2010-10-07 at 15:30 -0500, Justin M Wozniak wrote:
> > > >>>
> > > >>>> I meant release branch.  The valid branches could be hard-coded
> > > into the
> > > >>>> update.sh script.  The main guide would be the doc associated
> > > with the
> > > >>>> current version.  So right now, "main guide" would be aliased to
> > > 0.9 .
> > > >>>>
> > > >>>> On Thu, 7 Oct 2010, Sarah Kenny wrote:
> > > >>>>
> > > >>>>  was thinking of what justin said, "I propose we have one web
> > > site but
> > > >>>>> multiple docs/guides directories, all accessible from the
> > > docs/index.php
> > > >>>>> page.  Each of these would be associated with a branch"
> > > >>>>>
> > > >>>>> i was assuming that whatever branch(es) these were associated
> > > with, that
> > > >>>>> doc
> > > >>>>> would somehow need to make its way to a main guide that we are
> > > pointing
> > > >>>>> users to.
> > > >>>>>
> > > >>>>> ?
> > > >>>>>
> > > >>>>>
> > > >>>>> On Thu, Oct 7, 2010 at 3:17 PM, Mihael Hategan
> > > <hategan at mcs.anl.gov>
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>  On Thu, 2010-10-07 at 14:55 -0500, Sarah Kenny wrote:
> > > >>>>>>
> > > >>>>>>> so, in that case the 'main user doc' would be something
> > > *like*
> > > >>>>>>> http://www.ci.uchicago.edu/swift/docs10/index.php ?
> > > >>>>>>>
> > > >>>>>>> and THAT would include the updates from all the current
> > > branches
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>> define "all current branches". We have:
> > > >>>>>> 1. Release branches
> > > >>>>>> 2. Trunk
> > > >>>>>> 3. Development branches (which are transient entities and only
> > > there to
> > > >>>>>> make trunk's life easier).
> > > >>>>>>
> > > >>>>>>   merged into it once we do a release?
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> On Thu, Oct 7, 2010 at 2:42 PM, Mihael Hategan
> > > <hategan at mcs.anl.gov>
> > > >>>>>>> wrote:
> > > >>>>>>>        On Thu, 2010-10-07 at 14:39 -0500, Sarah Kenny wrote:
> > > >>>>>>>     > so, in this scenario, the changes to the doc that exist
> > > in
> > > >>>>>>>        each branch
> > > >>>>>>>     > are pushed to the main user doc when we do the release
> > > or am
> > > >>>>>>>        i missing
> > > >>>>>>>     > a step here?
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>        That or we really have no "main doc" and instead we
> > > link from
> > > >>>>>>>        every
> > > >>>>>>>        release. Though I feel odd about that.
> > > >>>>>>>
> > > >>>>>>>     >
> > > >>>>>>>     > On Thu, Oct 7, 2010 at 2:34 PM, Mihael Hategan
> > > >>>>>>>        <hategan at mcs.anl.gov>
> > > >>>>>>>     > wrote:
> > > >>>>>>>     >
> > > >>>>>>>     >         On Thu, 2010-10-07 at 14:28 -0500, Justin M
> > > Wozniak
> > > >>>>>>>        wrote:
> > > >>>>>>>     >      > On Thu, 7 Oct 2010, Sarah Kenny wrote:
> > > >>>>>>>     >      >
> > > >>>>>>>     >      >> On Thu, Oct 7, 2010 at 1:08 PM, Mihael Hategan
> > > >>>>>>>     >         <hategan at mcs.anl.gov> wrote:
> > > >>>>>>>     >      >>
> > > >>>>>>>     >      >>> Right. I think 1.0/4.1.7 should go out soon.
> > > >>>>>>>     >      >>
> > > >>>>>>>     >      >> ok, so i guess we should decide what 'soon'
> > > >>>>>>>        means ;) i am
> > > >>>>>>>     >         currently going
> > > >>>>>>>     >      >> thru the old bugs in bugzilla (at least trying
> > > >>>>>>>        to close
> > > >>>>>>>     >         out things that have
> > > >>>>>>>     >      >> been already fixed or are no-longer applicable,
> > > >>>>>>>        etc), but
> > > >>>>>>>     >         perhaps it would
> > > >>>>>>>     >      >> be good to determine if there are bigger issues
> > > >>>>>>>        outside of
> > > >>>>>>>     >         that that still
> > > >>>>>>>     >      >> need to be dealt with before we can put what
> > > >>>>>>>        we've got
> > > >>>>>>>     >         into a stable release
> > > >>>>>>>     >      >> and determine a time-frame...anything come to
> > > >>>>>>>        mind?
> > > >>>>>>>     >      >>
> > > >>>>>>>     >      >> as far as documentation...does it make sense
> > > for
> > > >>>>>>>        each
> > > >>>>>>>     >         branch to have a full
> > > >>>>>>>     >      >> copy of /ci/www/projects/swift under it which
> > > >>>>>>>        can then be
> > > >>>>>>>     >         merged with the
> > > >>>>>>>     >      >> main/live copy whenever the code is merged?
> > > >>>>>>>        admittedly, i
> > > >>>>>>>     >         know nothing about
> > > >>>>>>>     >      >> docbook, but from the standpoint of updating
> > > and
> > > >>>>>>>        merging
> > > >>>>>>>     >         this seems to make
> > > >>>>>>>     >      >> sense to me (though feel free to suggest
> > > another
> > > >>>>>>>        way :)
> > > >>>>>>>     >      >>
> > > >>>>>>>     >      >> ~sk
> > > >>>>>>>     >      >
> > > >>>>>>>     >      > I was looking at the update.sh script earlier
> > > >>>>>>>        today- I
> > > >>>>>>>     >         propose we have one
> > > >>>>>>>     >      > web site but multiple docs/guides directories,
> > > all
> > > >>>>>>>     >         accessible from the
> > > >>>>>>>     >      > docs/index.php page.  Each of these would be
> > > >>>>>>>        associated with
> > > >>>>>>>     >         a branch.
> > > >>>>>>>     >      > So, similar to the existing "Historical" section
> > > >>>>>>>        but for
> > > >>>>>>>     >         "future" branches
> > > >>>>>>>     >      > as well.  That would take a small modification
> > > to
> > > >>>>>>>        the
> > > >>>>>>>     >         update.sh script and
> > > >>>>>>>     >      > manual modification of the docs/index.php page
> > > for
> > > >>>>>>>        each
> > > >>>>>>>     >         version number.
> > > >>>>>>>     >      >
> > > >>>>>>>     >      > We may also want to have the feature changes
> > > (past
> > > >>>>>>>        and
> > > >>>>>>>     >         future version
> > > >>>>>>>     >      > numbers) available on that page but I think
> > > those
> > > >>>>>>>        can be
> > > >>>>>>>     >         plain text.
> > > >>>>>>>     >      > These could be pulled directly from SVN as well.
> > > >>>>>>>     >      >
> > > >>>>>>>     >
> > > >>>>>>>     >
> > > >>>>>>>     >         I agree. I generally believe that documentation
> > > >>>>>>>        should be kept
> > > >>>>>>>     >         in sync
> > > >>>>>>>     >         with releases (and I also think that the effort
> > > of
> > > >>>>>>>        doing so is
> > > >>>>>>>     >         minimal).
> > > >>>>>>>     >
> > > >>>>>>>     >
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>>
> > > >>>
> > > >> --
> > > >> Justin M Wozniak
> > > >>
> > > >
> > >
> > > --
> > > Justin M Wozniak
> > > _______________________________________________
> > > Swift-devel mailing list
> > > Swift-devel at ci.uchicago.edu
> > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> >
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101014/8784140d/attachment.html>

From jon.monette at gmail.com  Mon Oct 18 13:03:09 2010
From: jon.monette at gmail.com (Jonathan Monette)
Date: Mon, 18 Oct 2010 13:03:09 -0500
Subject: [Swift-devel] Memory in Swift
Message-ID: <4CBC8BDD.2020206@gmail.com>

  Hello,
     I have two questions.

     1) What is the maximum value for SWIFT_HEAP_MAX?  Is it the maximum 
memory available on the machine?  If I ask for 8Gb will the Java heap 
space be 8Gb?
     2) Assuming that the above question does not fix my problem, what 
can be done to fix memory problems in Swift?  How should a Swift script 
be programmed to avoid these issues or is how Swift handles memory not 
optimal.  I have a set of about 4100 images that will be stitched 
together to form a mosaic.  The size of this set is 8.1Gb.  Right now in 
my sites file I set SWIFT_HEAP_MAX to 4096M(4Gb) but I am still getting 
out of memory errors with the Java heap space.  Is Swift just not able 
to scale up to this size of data yet?

-- 
Jon

Computers are incredibly fast, accurate, and stupid. Human beings are incredibly slow, inaccurate, and brilliant. Together they are powerful beyond imagination.
- Albert Einstein



From wozniak at mcs.anl.gov  Mon Oct 18 13:21:45 2010
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Mon, 18 Oct 2010 13:21:45 -0500 (CDT)
Subject: [Swift-devel] Re: Memory in Swift
In-Reply-To: <4CBC8BDD.2020206@gmail.com>
References: <4CBC8BDD.2020206@gmail.com>
Message-ID: <alpine.DEB.2.00.1010181317210.2286@wozniak-desktop.mcs.anl.gov>

On Mon, 18 Oct 2010, Jonathan Monette wrote:

>    1) What is the maximum value for SWIFT_HEAP_MAX?  Is it the maximum 
> memory available on the machine?  If I ask for 8Gb will the Java heap space 
> be 8Gb?

In bin/swift, that value is passed directly to the Java VM, so it's 
implementation-specific.  What kind of machine are you on?

>    2) Assuming that the above question does not fix my problem, what can 
> be done to fix memory problems in Swift?  How should a Swift script be 
> programmed to avoid these issues or is how Swift handles memory not 
> optimal. I have a set of about 4100 images that will be stitched 
> together to form a mosaic.  The size of this set is 8.1Gb.  Right now in 
> my sites file I set SWIFT_HEAP_MAX to 4096M(4Gb) but I am still getting 
> out of memory errors with the Java heap space.  Is Swift just not able 
> to scale up to this size of data yet?

-- 
Justin M Wozniak


From jon.monette at gmail.com  Mon Oct 18 13:19:27 2010
From: jon.monette at gmail.com (Jonathan Monette)
Date: Mon, 18 Oct 2010 13:19:27 -0500
Subject: [Swift-devel] Re: Memory in Swift
In-Reply-To: <alpine.DEB.2.00.1010181317210.2286@wozniak-desktop.mcs.anl.gov>
References: <4CBC8BDD.2020206@gmail.com>
	<alpine.DEB.2.00.1010181317210.2286@wozniak-desktop.mcs.anl.gov>
Message-ID: <4CBC8FAF.1000905@gmail.com>

  I am on PADS.

On 10/18/2010 01:21 PM, Justin M Wozniak wrote:
> On Mon, 18 Oct 2010, Jonathan Monette wrote:
>
>>    1) What is the maximum value for SWIFT_HEAP_MAX?  Is it the 
>> maximum memory available on the machine?  If I ask for 8Gb will the 
>> Java heap space be 8Gb?
>
> In bin/swift, that value is passed directly to the Java VM, so it's 
> implementation-specific.  What kind of machine are you on?
>
>>    2) Assuming that the above question does not fix my problem, what 
>> can be done to fix memory problems in Swift?  How should a Swift 
>> script be programmed to avoid these issues or is how Swift handles 
>> memory not optimal. I have a set of about 4100 images that will be 
>> stitched together to form a mosaic.  The size of this set is 8.1Gb.  
>> Right now in my sites file I set SWIFT_HEAP_MAX to 4096M(4Gb) but I 
>> am still getting out of memory errors with the Java heap space.  Is 
>> Swift just not able to scale up to this size of data yet?
>

-- 
Jon

Computers are incredibly fast, accurate, and stupid. Human beings are incredibly slow, inaccurate, and brilliant. Together they are powerful beyond imagination.
- Albert Einstein



From wozniak at mcs.anl.gov  Mon Oct 18 16:36:23 2010
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Mon, 18 Oct 2010 16:36:23 -0500 (CDT)
Subject: [Swift-devel] Re: Memory in Swift
In-Reply-To: <4CBC8FAF.1000905@gmail.com>
References: <4CBC8BDD.2020206@gmail.com>
	<alpine.DEB.2.00.1010181317210.2286@wozniak-desktop.mcs.anl.gov>
	<4CBC8FAF.1000905@gmail.com>
Message-ID: <alpine.DEB.2.00.1010181635570.2286@wozniak-desktop.mcs.anl.gov>


What do you get if you set it to 8192M?

On Mon, 18 Oct 2010, Jonathan Monette wrote:

> I am on PADS.
>
> On 10/18/2010 01:21 PM, Justin M Wozniak wrote:
>> On Mon, 18 Oct 2010, Jonathan Monette wrote:
>>
>>>    1) What is the maximum value for SWIFT_HEAP_MAX?  Is it the maximum 
>>> memory available on the machine?  If I ask for 8Gb will the Java heap 
>>> space be 8Gb?
>> 
>> In bin/swift, that value is passed directly to the Java VM, so it's 
>> implementation-specific.  What kind of machine are you on?
>>
>>>    2) Assuming that the above question does not fix my problem, what can 
>>> be done to fix memory problems in Swift?  How should a Swift script be 
>>> programmed to avoid these issues or is how Swift handles memory not 
>>> optimal. I have a set of about 4100 images that will be stitched together 
>>> to form a mosaic.  The size of this set is 8.1Gb.  Right now in my sites 
>>> file I set SWIFT_HEAP_MAX to 4096M(4Gb) but I am still getting out of 
>>> memory errors with the Java heap space.  Is Swift just not able to scale 
>>> up to this size of data yet?
>> 
>
>

-- 
Justin M Wozniak


From jon.monette at gmail.com  Mon Oct 18 16:46:14 2010
From: jon.monette at gmail.com (Jonathan Monette)
Date: Mon, 18 Oct 2010 16:46:14 -0500
Subject: [Swift-devel] Re: Memory in Swift
In-Reply-To: <alpine.DEB.2.00.1010181635570.2286@wozniak-desktop.mcs.anl.gov>
References: <4CBC8BDD.2020206@gmail.com>
	<alpine.DEB.2.00.1010181317210.2286@wozniak-desktop.mcs.anl.gov>
	<4CBC8FAF.1000905@gmail.com>
	<alpine.DEB.2.00.1010181635570.2286@wozniak-desktop.mcs.anl.gov>
Message-ID: <4CBCC026.8090708@gmail.com>

  jonmon   22946 54.6  1.2 788800 318496 pts/17  Sl+  16:42   0:39 java 
-Xmx256M

I am assuming that this means the heap space is 256 Megabytes.  Here is 
how I set the SWIFT_HEAP_MAX variable:

<profile key="SWIFT_HEAP_MAX" namespace="env">8192M</profile>

Is this how you would set environment variables in the sites.xml file?  
I want to make sure the variable is also set for the worker nodes.

On 10/18/2010 04:36 PM, Justin M Wozniak wrote:
>
> What do you get if you set it to 8192M?
>
> On Mon, 18 Oct 2010, Jonathan Monette wrote:
>
>> I am on PADS.
>>
>> On 10/18/2010 01:21 PM, Justin M Wozniak wrote:
>>> On Mon, 18 Oct 2010, Jonathan Monette wrote:
>>>
>>>>    1) What is the maximum value for SWIFT_HEAP_MAX?  Is it the 
>>>> maximum memory available on the machine?  If I ask for 8Gb will the 
>>>> Java heap space be 8Gb?
>>>
>>> In bin/swift, that value is passed directly to the Java VM, so it's 
>>> implementation-specific.  What kind of machine are you on?
>>>
>>>>    2) Assuming that the above question does not fix my problem, 
>>>> what can be done to fix memory problems in Swift?  How should a 
>>>> Swift script be programmed to avoid these issues or is how Swift 
>>>> handles memory not optimal. I have a set of about 4100 images that 
>>>> will be stitched together to form a mosaic.  The size of this set 
>>>> is 8.1Gb.  Right now in my sites file I set SWIFT_HEAP_MAX to 
>>>> 4096M(4Gb) but I am still getting out of memory errors with the 
>>>> Java heap space.  Is Swift just not able to scale up to this size 
>>>> of data yet?
>>>
>>
>>
>

-- 
Jon

Computers are incredibly fast, accurate, and stupid. Human beings are incredibly slow, inaccurate, and brilliant. Together they are powerful beyond imagination.
- Albert Einstein



From wozniak at mcs.anl.gov  Mon Oct 18 17:14:52 2010
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Mon, 18 Oct 2010 17:14:52 -0500 (CDT)
Subject: [Swift-devel] Re: Memory in Swift
In-Reply-To: <4CBCC026.8090708@gmail.com>
References: <4CBC8BDD.2020206@gmail.com>
	<alpine.DEB.2.00.1010181317210.2286@wozniak-desktop.mcs.anl.gov>
	<4CBC8FAF.1000905@gmail.com>
	<alpine.DEB.2.00.1010181635570.2286@wozniak-desktop.mcs.anl.gov>
	<4CBCC026.8090708@gmail.com>
Message-ID: <alpine.DEB.2.00.1010181709460.2286@wozniak-desktop.mcs.anl.gov>


Actually, to set SWIFT_HEAP_MAX for Swift you should export it in the 
environment in which you run bin/swift .  So, in the shell you could do:

SWIFT_HEAP_MAX=8192M swift ...

The namespace env is there to let you set the environment for the remote 
user job.

On Mon, 18 Oct 2010, Jonathan Monette wrote:

> jonmon   22946 54.6  1.2 788800 318496 pts/17  Sl+  16:42   0:39 java 
> -Xmx256M
>
> I am assuming that this means the heap space is 256 Megabytes.  Here is how I 
> set the SWIFT_HEAP_MAX variable:
>
> <profile key="SWIFT_HEAP_MAX" namespace="env">8192M</profile>
>
> Is this how you would set environment variables in the sites.xml file?  I 
> want to make sure the variable is also set for the worker nodes.
>
> On 10/18/2010 04:36 PM, Justin M Wozniak wrote:
>> 
>> What do you get if you set it to 8192M?
>> 
>> On Mon, 18 Oct 2010, Jonathan Monette wrote:
>> 
>>> I am on PADS.
>>> 
>>> On 10/18/2010 01:21 PM, Justin M Wozniak wrote:
>>>> On Mon, 18 Oct 2010, Jonathan Monette wrote:
>>>>
>>>>>    1) What is the maximum value for SWIFT_HEAP_MAX?  Is it the maximum 
>>>>> memory available on the machine?  If I ask for 8Gb will the Java heap 
>>>>> space be 8Gb?
>>>> 
>>>> In bin/swift, that value is passed directly to the Java VM, so it's 
>>>> implementation-specific.  What kind of machine are you on?
>>>>
>>>>>    2) Assuming that the above question does not fix my problem, what can 
>>>>> be done to fix memory problems in Swift?  How should a Swift script be 
>>>>> programmed to avoid these issues or is how Swift handles memory not 
>>>>> optimal. I have a set of about 4100 images that will be stitched 
>>>>> together to form a mosaic.  The size of this set is 8.1Gb.  Right now in 
>>>>> my sites file I set SWIFT_HEAP_MAX to 4096M(4Gb) but I am still getting 
>>>>> out of memory errors with the Java heap space.  Is Swift just not able 
>>>>> to scale up to this size of data yet?
>>>> 
>>> 
>>> 
>> 
>
>

-- 
Justin M Wozniak


From jon.monette at gmail.com  Mon Oct 18 17:18:49 2010
From: jon.monette at gmail.com (jon.monette at gmail.com)
Date: Mon, 18 Oct 2010 22:18:49 +0000
Subject: [Swift-devel] Re: Memory in Swift
In-Reply-To: <alpine.DEB.2.00.1010181709460.2286@wozniak-desktop.mcs.anl.gov>
References: <4CBC8BDD.2020206@gmail.com>
	<alpine.DEB.2.00.1010181317210.2286@wozniak-desktop.mcs.anl.gov>
	<4CBC8FAF.1000905@gmail.com>
	<alpine.DEB.2.00.1010181635570.2286@wozniak-desktop.mcs.anl.gov>
	<4CBCC026.8090708@gmail.com><alpine.DEB.2.00.1010181709460.2286@wozniak-desktop.mcs.anl.gov>
Message-ID: <655791171-1287440330-cardhu_decombobulator_blackberry.rim.net-1940688138-@bda090.bisx.prod.on.blackberry>

Ok. so I only have to make sure it is set in the shell in which I call swift? It doesn't have to be set for the remote jobs?
Sent on the Sprint? Now Network from my BlackBerry?

-----Original Message-----
From: Justin M Wozniak <wozniak at mcs.anl.gov>
Date: Mon, 18 Oct 2010 17:14:52 
To: Jonathan Monette<jon.monette at gmail.com>
Cc: Swift Devel<swift-devel at ci.uchicago.edu>
Subject: Re: Memory in Swift


Actually, to set SWIFT_HEAP_MAX for Swift you should export it in the 
environment in which you run bin/swift .  So, in the shell you could do:

SWIFT_HEAP_MAX=8192M swift ...

The namespace env is there to let you set the environment for the remote 
user job.

On Mon, 18 Oct 2010, Jonathan Monette wrote:

> jonmon   22946 54.6  1.2 788800 318496 pts/17  Sl+  16:42   0:39 java 
> -Xmx256M
>
> I am assuming that this means the heap space is 256 Megabytes.  Here is how I 
> set the SWIFT_HEAP_MAX variable:
>
> <profile key="SWIFT_HEAP_MAX" namespace="env">8192M</profile>
>
> Is this how you would set environment variables in the sites.xml file?  I 
> want to make sure the variable is also set for the worker nodes.
>
> On 10/18/2010 04:36 PM, Justin M Wozniak wrote:
>> 
>> What do you get if you set it to 8192M?
>> 
>> On Mon, 18 Oct 2010, Jonathan Monette wrote:
>> 
>>> I am on PADS.
>>> 
>>> On 10/18/2010 01:21 PM, Justin M Wozniak wrote:
>>>> On Mon, 18 Oct 2010, Jonathan Monette wrote:
>>>>
>>>>>    1) What is the maximum value for SWIFT_HEAP_MAX?  Is it the maximum 
>>>>> memory available on the machine?  If I ask for 8Gb will the Java heap 
>>>>> space be 8Gb?
>>>> 
>>>> In bin/swift, that value is passed directly to the Java VM, so it's 
>>>> implementation-specific.  What kind of machine are you on?
>>>>
>>>>>    2) Assuming that the above question does not fix my problem, what can 
>>>>> be done to fix memory problems in Swift?  How should a Swift script be 
>>>>> programmed to avoid these issues or is how Swift handles memory not 
>>>>> optimal. I have a set of about 4100 images that will be stitched 
>>>>> together to form a mosaic.  The size of this set is 8.1Gb.  Right now in 
>>>>> my sites file I set SWIFT_HEAP_MAX to 4096M(4Gb) but I am still getting 
>>>>> out of memory errors with the Java heap space.  Is Swift just not able 
>>>>> to scale up to this size of data yet?
>>>> 
>>> 
>>> 
>> 
>
>

-- 
Justin M Wozniak

From jon.monette at gmail.com  Mon Oct 18 20:22:50 2010
From: jon.monette at gmail.com (Jonathan Monette)
Date: Mon, 18 Oct 2010 20:22:50 -0500
Subject: [Swift-devel] Re: Memory in Swift
In-Reply-To: <alpine.DEB.2.00.1010181709460.2286@wozniak-desktop.mcs.anl.gov>
References: <4CBC8BDD.2020206@gmail.com>
	<alpine.DEB.2.00.1010181317210.2286@wozniak-desktop.mcs.anl.gov>
	<4CBC8FAF.1000905@gmail.com>
	<alpine.DEB.2.00.1010181635570.2286@wozniak-desktop.mcs.anl.gov>
	<4CBCC026.8090708@gmail.com>
	<alpine.DEB.2.00.1010181709460.2286@wozniak-desktop.mcs.anl.gov>
Message-ID: <4CBCF2EA.3020005@gmail.com>

  Ok.  That seemed to fix my out of memory problem.

On 10/18/2010 05:14 PM, Justin M Wozniak wrote:
>
> Actually, to set SWIFT_HEAP_MAX for Swift you should export it in the 
> environment in which you run bin/swift .  So, in the shell you could do:
>
> SWIFT_HEAP_MAX=8192M swift ...
>
> The namespace env is there to let you set the environment for the 
> remote user job.
>
> On Mon, 18 Oct 2010, Jonathan Monette wrote:
>
>> jonmon   22946 54.6  1.2 788800 318496 pts/17  Sl+  16:42   0:39 java 
>> -Xmx256M
>>
>> I am assuming that this means the heap space is 256 Megabytes.  Here 
>> is how I set the SWIFT_HEAP_MAX variable:
>>
>> <profile key="SWIFT_HEAP_MAX" namespace="env">8192M</profile>
>>
>> Is this how you would set environment variables in the sites.xml 
>> file?  I want to make sure the variable is also set for the worker 
>> nodes.
>>
>> On 10/18/2010 04:36 PM, Justin M Wozniak wrote:
>>>
>>> What do you get if you set it to 8192M?
>>>
>>> On Mon, 18 Oct 2010, Jonathan Monette wrote:
>>>
>>>> I am on PADS.
>>>>
>>>> On 10/18/2010 01:21 PM, Justin M Wozniak wrote:
>>>>> On Mon, 18 Oct 2010, Jonathan Monette wrote:
>>>>>
>>>>>>    1) What is the maximum value for SWIFT_HEAP_MAX?  Is it the 
>>>>>> maximum memory available on the machine?  If I ask for 8Gb will 
>>>>>> the Java heap space be 8Gb?
>>>>>
>>>>> In bin/swift, that value is passed directly to the Java VM, so 
>>>>> it's implementation-specific.  What kind of machine are you on?
>>>>>
>>>>>>    2) Assuming that the above question does not fix my problem, 
>>>>>> what can be done to fix memory problems in Swift?  How should a 
>>>>>> Swift script be programmed to avoid these issues or is how Swift 
>>>>>> handles memory not optimal. I have a set of about 4100 images 
>>>>>> that will be stitched together to form a mosaic.  The size of 
>>>>>> this set is 8.1Gb.  Right now in my sites file I set 
>>>>>> SWIFT_HEAP_MAX to 4096M(4Gb) but I am still getting out of memory 
>>>>>> errors with the Java heap space.  Is Swift just not able to scale 
>>>>>> up to this size of data yet?
>>>>>
>>>>
>>>>
>>>
>>
>>
>

-- 
Jon

Computers are incredibly fast, accurate, and stupid. Human beings are incredibly slow, inaccurate, and brilliant. Together they are powerful beyond imagination.
- Albert Einstein



From hategan at mcs.anl.gov  Mon Oct 18 21:41:25 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Mon, 18 Oct 2010 19:41:25 -0700
Subject: [Swift-devel] Re: Memory in Swift
In-Reply-To: <4CBCF2EA.3020005@gmail.com>
References: <4CBC8BDD.2020206@gmail.com>
	<alpine.DEB.2.00.1010181317210.2286@wozniak-desktop.mcs.anl.gov>
	<4CBC8FAF.1000905@gmail.com>
	<alpine.DEB.2.00.1010181635570.2286@wozniak-desktop.mcs.anl.gov>
	<4CBCC026.8090708@gmail.com>
	<alpine.DEB.2.00.1010181709460.2286@wozniak-desktop.mcs.anl.gov>
	<4CBCF2EA.3020005@gmail.com>
Message-ID: <1287456085.5764.4.camel@blabla2.none>

A side note: I think that 32 bit JVMs have a limit of around 3GB (the
limit for a 32 bit process).

You should also be wary of the following possibility: when a process is
forked in linux, it first inherits the parent's heap size (so total VM
is double of the original process). That gets quickly dropped by the
System.exec() implementation as soon as the process is started, but for
a small window of time (and with multiple concurrent forks the problem
may get worse), there is a chance that you will run out of virtual
memory.

Mihael

On Mon, 2010-10-18 at 20:22 -0500, Jonathan Monette wrote:
> Ok.  That seemed to fix my out of memory problem.
> 
> On 10/18/2010 05:14 PM, Justin M Wozniak wrote:
> >
> > Actually, to set SWIFT_HEAP_MAX for Swift you should export it in the 
> > environment in which you run bin/swift .  So, in the shell you could do:
> >
> > SWIFT_HEAP_MAX=8192M swift ...
> >
> > The namespace env is there to let you set the environment for the 
> > remote user job.
> >
> > On Mon, 18 Oct 2010, Jonathan Monette wrote:
> >
> >> jonmon   22946 54.6  1.2 788800 318496 pts/17  Sl+  16:42   0:39 java 
> >> -Xmx256M
> >>
> >> I am assuming that this means the heap space is 256 Megabytes.  Here 
> >> is how I set the SWIFT_HEAP_MAX variable:
> >>
> >> <profile key="SWIFT_HEAP_MAX" namespace="env">8192M</profile>
> >>
> >> Is this how you would set environment variables in the sites.xml 
> >> file?  I want to make sure the variable is also set for the worker 
> >> nodes.
> >>
> >> On 10/18/2010 04:36 PM, Justin M Wozniak wrote:
> >>>
> >>> What do you get if you set it to 8192M?
> >>>
> >>> On Mon, 18 Oct 2010, Jonathan Monette wrote:
> >>>
> >>>> I am on PADS.
> >>>>
> >>>> On 10/18/2010 01:21 PM, Justin M Wozniak wrote:
> >>>>> On Mon, 18 Oct 2010, Jonathan Monette wrote:
> >>>>>
> >>>>>>    1) What is the maximum value for SWIFT_HEAP_MAX?  Is it the 
> >>>>>> maximum memory available on the machine?  If I ask for 8Gb will 
> >>>>>> the Java heap space be 8Gb?
> >>>>>
> >>>>> In bin/swift, that value is passed directly to the Java VM, so 
> >>>>> it's implementation-specific.  What kind of machine are you on?
> >>>>>
> >>>>>>    2) Assuming that the above question does not fix my problem, 
> >>>>>> what can be done to fix memory problems in Swift?  How should a 
> >>>>>> Swift script be programmed to avoid these issues or is how Swift 
> >>>>>> handles memory not optimal. I have a set of about 4100 images 
> >>>>>> that will be stitched together to form a mosaic.  The size of 
> >>>>>> this set is 8.1Gb.  Right now in my sites file I set 
> >>>>>> SWIFT_HEAP_MAX to 4096M(4Gb) but I am still getting out of memory 
> >>>>>> errors with the Java heap space.  Is Swift just not able to scale 
> >>>>>> up to this size of data yet?
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >
> 




From bugzilla-daemon at mcs.anl.gov  Tue Oct 19 13:59:57 2010
From: bugzilla-daemon at mcs.anl.gov (bugzilla-daemon at mcs.anl.gov)
Date: Tue, 19 Oct 2010 13:59:57 -0500 (CDT)
Subject: [Swift-devel] [Bug 112] error reporting in procedure declarations
In-Reply-To: <bug-112-21@http.bugzilla.mcs.anl.gov/swift/>
References: <bug-112-21@http.bugzilla.mcs.anl.gov/swift/>
Message-ID: <20101019185958.048C12BFAF@wind.mcs.anl.gov>

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=112





--- Comment #2 from skenny <skenny at uchicago.edu>  2010-10-19 13:59:57 ---
added a test for this in:
tests/language/should-not-work/118-keyword-misuse-stdout.swift

-- 
Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.
You are watching the assignee of the bug.
You are watching the reporter.


From iraicu at cs.uchicago.edu  Tue Oct 19 21:51:16 2010
From: iraicu at cs.uchicago.edu (Ioan Raicu)
Date: Tue, 19 Oct 2010 21:51:16 -0500
Subject: [Swift-devel] Emerging Research and Development session at CloudCom
 2010 November 30-December 3, 2010
Message-ID: <4CBE5924.4030606@cs.uchicago.edu>

The main track track(207 papers submitted, 48 accepted), workshops and 
tutorials schedules are already posted 
http://salsahpc.indiana.edu/CloudCom2010/schedule.html but Emerging 
Research and Development session is still open. Of course you could just 
attend!

*IEEE 2nd International Conference on Cloud Computing Technology and Science
November 30-December 3, 2010
Indianapolis, Indiana
*http://2010.cloudcom.org/
*Earlybird registration through October 31. *


The IEEE CloudCom 2010 conference welcomes proposals for an *Emerging 
Research and Development session *that includes student projects, new 
cloud commercial products, hot topics and wild and crazy ideas. Each 
accepted project will be displayed or demonstrated during the 
poster/exhibit/demo session in the evening of Wednesday December 1, 2010.

All audiences are welcome to submit their research material, including 
all levels of students, researchers and those from corporations dealing 
with cloud technologies. All three types of material will be displayed 
in a single space during the poster reception. (Posters accepted to the 
IEEE CloudCom 2010 technical program of the will be displayed in a 
separate section.)

This is a perfect opportunity to get your emerging research seen by the 
international Cloud Community!
*
Emerging Research Submission* *Guidelines*
Proposals for the Emerging Research and Development Session should be 
submitted for review in the form of an extended abstract with the 
following submission requirements:

    * Up to 2 Pages in Length
    * Single-spaced, double column text in 10-point font
    * 8.5 x 11 inch pages
    * PDF Formatted

The abstracts should clearly present the work described, including the 
problem statement, previous work as well as contribution of work. There 
should be a clear relevance to clouds in work. Each abstract must 
contain a section that describes the overall structure of the poster or 
display; the authors are strongly encouraged to present examples of 
graphs and illustrations that will appear on the poster. In addition, 
each abstract proposing an exhibition or a demo should contain a 
description of facilities that the authors expect to be provided. Each 
abstract should describe category in which work falls -- *student 
projects*, *commercial products*, *hot topics* or *wild and crazy ideas*.

Accepted abstracts will be posted on the conference Web site. 
Additionally, a video crew will record selected posters, demos, and 
exhibits. Authors of accepted extended abstracts will be required to 
sign a release for the recording of these videos. Accepted presenters 
will be required to register for the full conference (no one-day 
registrations accepted for presenters). Accepted presenters will be 
granted the earlybird conference rate even if they are notified of 
acceptance after the earlybird cutoff date.

Visit http://2010.cloudcom.org/ for complete submission guidelines.


-- 
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor
=================================================================
Computer Science Department
Illinois Institute of Technology
10 W. 31st Street
Stuart Building, Room 237D
Chicago, IL 60616
=================================================================
Cel:    1-847-722-0876
Office: 1-312-567-5704
Email:  iraicu at cs.iit.edu
Web:    http://www.cs.iit.edu/~iraicu/
=================================================================
=================================================================




-- 
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor
=================================================================
Computer Science Department
Illinois Institute of Technology
10 W. 31st Street
Stuart Building, Room 237D
Chicago, IL 60616
=================================================================
Cel:    1-847-722-0876
Office: 1-312-567-5704
Email:  iraicu at cs.iit.edu
Web:    http://www.cs.iit.edu/~iraicu/
=================================================================
=================================================================


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101019/14316557/attachment.html>

From jon.monette at gmail.com  Wed Oct 20 10:39:19 2010
From: jon.monette at gmail.com (Jonathan Monette)
Date: Wed, 20 Oct 2010 10:39:19 -0500
Subject: [Swift-devel] Standard Swift coaster behavior doesnt work well
	for sporadic jobs
In-Reply-To: <57986955.982201286845836821.JavaMail.root@zimbra.anl.gov>
References: <57986955.982201286845836821.JavaMail.root@zimbra.anl.gov>
Message-ID: <4CBF0D27.80302@gmail.com>

  Has this problem been fixed?  I am still experiencing hanging in my 
scripts and it seems that the jobs are submitted but never executed.  I 
see that the stage out is finished, the jobs are submitted, and then the 
coaster heartbeat in the logfile.

On 10/11/10 8:10 PM, Michael Wilde wrote:
> ----- "Jonathan Monette"<jon.monette at gmail.com>  wrote:
>
>> Wouldn't coasters just re-submit jobs if there are no workers
>> available to process them?
> Thats certainly the desired behavior for the default "automatic" mode, but it doesnt appear to be working that way - unless Ive broken it with a local mod.
>
> - Mike
>
>
>> My Montage stuff is under the assumption
>>
>> that coasters will submit more workers if they all time out.  This
>> maybe
>> why my stuff was hanging before.  Not entirely for sure since I am
>> working on another problem.
>
>> On 10/11/2010 03:12 PM, wilde at mcs.anl.gov wrote:
>>> Mihael, Justin, does the following sound like a likely coaster
>> issue:
>>> When using the standard Swift coaster code (not passive or
>> persistent), if I have a job that runs at the start, and then there is
>> a long delay before the next job, such that the coaster worker times
>> out, then the coaster scheduler doesnt think that there is a valid
>> block into which the job can fit, and Swift just hangs, with the job
>> in submitted state but never getting assigned to a block.
>>> The behavior seems similar to what you see when you try to run a job
>> that doesnt fit into any block that you have defined using the
>> coasters sites.xml parameters: in that case, too, Swift just hangs.
>>> Both of these situations (whether or not they are indeed due to the
>> same algorithmic issue) seem to be problems that we need to address.
>> In the first case (which is my immediate and more important problem)
>> you can see a log for the problem in ~wilde/swift-rserver-hangs, along
>> with the swift script, tc, sites, and properties  file(cf).
>>> Thanks,
>>>
>>> Mike
>>>
>> -- 
>> Jon
>>
>> Computers are incredibly fast, accurate, and stupid. Human beings are
>> incredibly slow, inaccurate, and brilliant. Together they are powerful
>> beyond imagination.
>> - Albert Einstein

-- 
Jon

Computers are incredibly fast, accurate, and stupid. Human beings are incredibly slow, inaccurate, and brilliant. Together they are powerful beyond imagination.
- Albert Einstein



From hategan at mcs.anl.gov  Wed Oct 20 18:58:09 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 20 Oct 2010 16:58:09 -0700
Subject: [Swift-devel] Standard Swift coaster behavior doesnt work well
	for sporadic jobs
In-Reply-To: <4CBF0D27.80302@gmail.com>
References: <57986955.982201286845836821.JavaMail.root@zimbra.anl.gov>
	<4CBF0D27.80302@gmail.com>
Message-ID: <1287619089.13330.2.camel@blabla2.none>

I haven't fixed it.

Is there a way to reproduce this nicely?

Mihael

On Wed, 2010-10-20 at 10:39 -0500, Jonathan Monette wrote:
> Has this problem been fixed?  I am still experiencing hanging in my 
> scripts and it seems that the jobs are submitted but never executed.  I 
> see that the stage out is finished, the jobs are submitted, and then the 
> coaster heartbeat in the logfile.
> 
> On 10/11/10 8:10 PM, Michael Wilde wrote:
> > ----- "Jonathan Monette"<jon.monette at gmail.com>  wrote:
> >
> >> Wouldn't coasters just re-submit jobs if there are no workers
> >> available to process them?
> > Thats certainly the desired behavior for the default "automatic" mode, but it doesnt appear to be working that way - unless Ive broken it with a local mod.
> >
> > - Mike
> >
> >
> >> My Montage stuff is under the assumption
> >>
> >> that coasters will submit more workers if they all time out.  This
> >> maybe
> >> why my stuff was hanging before.  Not entirely for sure since I am
> >> working on another problem.
> >
> >> On 10/11/2010 03:12 PM, wilde at mcs.anl.gov wrote:
> >>> Mihael, Justin, does the following sound like a likely coaster
> >> issue:
> >>> When using the standard Swift coaster code (not passive or
> >> persistent), if I have a job that runs at the start, and then there is
> >> a long delay before the next job, such that the coaster worker times
> >> out, then the coaster scheduler doesnt think that there is a valid
> >> block into which the job can fit, and Swift just hangs, with the job
> >> in submitted state but never getting assigned to a block.
> >>> The behavior seems similar to what you see when you try to run a job
> >> that doesnt fit into any block that you have defined using the
> >> coasters sites.xml parameters: in that case, too, Swift just hangs.
> >>> Both of these situations (whether or not they are indeed due to the
> >> same algorithmic issue) seem to be problems that we need to address.
> >> In the first case (which is my immediate and more important problem)
> >> you can see a log for the problem in ~wilde/swift-rserver-hangs, along
> >> with the swift script, tc, sites, and properties  file(cf).
> >>> Thanks,
> >>>
> >>> Mike
> >>>
> >> -- 
> >> Jon
> >>
> >> Computers are incredibly fast, accurate, and stupid. Human beings are
> >> incredibly slow, inaccurate, and brilliant. Together they are powerful
> >> beyond imagination.
> >> - Albert Einstein
> 




From jon.monette at gmail.com  Wed Oct 20 19:00:55 2010
From: jon.monette at gmail.com (jon.monette at gmail.com)
Date: Thu, 21 Oct 2010 00:00:55 +0000
Subject: [Swift-devel] Standard Swift coaster behavior doesnt work wellfor
	sporadic jobs
In-Reply-To: <1287619089.13330.2.camel@blabla2.none>
References: <57986955.982201286845836821.JavaMail.root@zimbra.anl.gov>
	<4CBF0D27.80302@gmail.com><1287619089.13330.2.camel@blabla2.none>
Message-ID: <1268212145-1287619257-cardhu_decombobulator_blackberry.rim.net-1673550369-@bda090.bisx.prod.on.blackberry>

Ok. I can try to put together a script that does it. But I think it just need to be a script in which between two jobs that are submitted to a site there is a long time so all the workers time out. 
Sent on the Sprint? Now Network from my BlackBerry?

-----Original Message-----
From: Mihael Hategan <hategan at mcs.anl.gov>
Date: Wed, 20 Oct 2010 16:58:09 
To: Jonathan Monette<jon.monette at gmail.com>
Cc: Michael Wilde<wilde at mcs.anl.gov>; Justin Wozniak<wozniak at mcs.anl.gov>; Swift Devel<swift-devel at ci.uchicago.edu>
Subject: Re: [Swift-devel] Standard Swift coaster behavior doesnt work well
 for sporadic jobs

I haven't fixed it.

Is there a way to reproduce this nicely?

Mihael

On Wed, 2010-10-20 at 10:39 -0500, Jonathan Monette wrote:
> Has this problem been fixed?  I am still experiencing hanging in my 
> scripts and it seems that the jobs are submitted but never executed.  I 
> see that the stage out is finished, the jobs are submitted, and then the 
> coaster heartbeat in the logfile.
> 
> On 10/11/10 8:10 PM, Michael Wilde wrote:
> > ----- "Jonathan Monette"<jon.monette at gmail.com>  wrote:
> >
> >> Wouldn't coasters just re-submit jobs if there are no workers
> >> available to process them?
> > Thats certainly the desired behavior for the default "automatic" mode, but it doesnt appear to be working that way - unless Ive broken it with a local mod.
> >
> > - Mike
> >
> >
> >> My Montage stuff is under the assumption
> >>
> >> that coasters will submit more workers if they all time out.  This
> >> maybe
> >> why my stuff was hanging before.  Not entirely for sure since I am
> >> working on another problem.
> >
> >> On 10/11/2010 03:12 PM, wilde at mcs.anl.gov wrote:
> >>> Mihael, Justin, does the following sound like a likely coaster
> >> issue:
> >>> When using the standard Swift coaster code (not passive or
> >> persistent), if I have a job that runs at the start, and then there is
> >> a long delay before the next job, such that the coaster worker times
> >> out, then the coaster scheduler doesnt think that there is a valid
> >> block into which the job can fit, and Swift just hangs, with the job
> >> in submitted state but never getting assigned to a block.
> >>> The behavior seems similar to what you see when you try to run a job
> >> that doesnt fit into any block that you have defined using the
> >> coasters sites.xml parameters: in that case, too, Swift just hangs.
> >>> Both of these situations (whether or not they are indeed due to the
> >> same algorithmic issue) seem to be problems that we need to address.
> >> In the first case (which is my immediate and more important problem)
> >> you can see a log for the problem in ~wilde/swift-rserver-hangs, along
> >> with the swift script, tc, sites, and properties  file(cf).
> >>> Thanks,
> >>>
> >>> Mike
> >>>
> >> -- 
> >> Jon
> >>
> >> Computers are incredibly fast, accurate, and stupid. Human beings are
> >> incredibly slow, inaccurate, and brilliant. Together they are powerful
> >> beyond imagination.
> >> - Albert Einstein
> 



From hategan at mcs.anl.gov  Thu Oct 21 01:31:25 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 20 Oct 2010 23:31:25 -0700
Subject: [Swift-devel] Standard Swift coaster behavior doesnt work
	wellfor sporadic jobs
In-Reply-To: <1268212145-1287619257-cardhu_decombobulator_blackberry.rim.net-1673550369-@bda090.bisx.prod.on.blackberry>
References: <57986955.982201286845836821.JavaMail.root@zimbra.anl.gov>
	<4CBF0D27.80302@gmail.com><1287619089.13330.2.camel@blabla2.none>
	<1268212145-1287619257-cardhu_decombobulator_blackberry.rim.net-1673550369-@bda090.bisx.prod.on.blackberry>
Message-ID: <1287642685.15118.46.camel@blabla2.none>

On Thu, 2010-10-21 at 00:00 +0000, jon.monette at gmail.com wrote:
> Ok. I can try to put together a script that does it. But I think it just need to be a script in which between two jobs that are submitted to a site there is a long time so all the workers time out. 

I can probably do that with a local sleep sandwiched between two coaster
jobs. So don't worry about it.

Mihael




From skenny at uchicago.edu  Thu Oct 21 11:43:08 2010
From: skenny at uchicago.edu (Sarah Kenny)
Date: Thu, 21 Oct 2010 11:43:08 -0500
Subject: [Swift-devel] post commit email notify
Message-ID: <AANLkTiksjQ0JbHpfVdi8XdArj-GW=LLfwYCVVHDZHRBW@mail.gmail.com>

would anyone have an objection to having an email notification sent to
swift-devel when commits are made to trunk? i thought this might be helpful,
but i'm not sure if others think this might be overkill (?)

~sk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101021/7eb870c9/attachment.html>

From wilde at mcs.anl.gov  Thu Oct 21 11:58:10 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 21 Oct 2010 10:58:10 -0600 (GMT-06:00)
Subject: [Swift-devel] post commit email notify
In-Reply-To: <AANLkTiksjQ0JbHpfVdi8XdArj-GW=LLfwYCVVHDZHRBW@mail.gmail.com>
Message-ID: <1473659805.199701287680290433.JavaMail.root@zimbra.anl.gov>

I think Im OK with that. Or, to a new list swift-commits? How is the being routed currently? To the Argonne lists? 

I think that Im getting some kind of list approval bounces for some of this traffic that I'm currently ignoring, and perhaps thats causing a problem. 

- Mike 

----- "Sarah Kenny" <skenny at uchicago.edu> wrote: 
> would anyone have an objection to having an email notification sent to swift-devel when commits are made to trunk? i thought this might be helpful, but i'm not sure if others think this might be overkill (?) 
> 
> ~sk 
> 
> _______________________________________________ Swift-devel mailing list Swift-devel at ci.uchicago.edu http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel 

-- 
Michael Wilde 
Computation Institute, University of Chicago 
Mathematics and Computer Science Division 
Argonne National Laboratory 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101021/1dedc032/attachment.html>

From skenny at uchicago.edu  Thu Oct 21 12:02:36 2010
From: skenny at uchicago.edu (Sarah Kenny)
Date: Thu, 21 Oct 2010 12:02:36 -0500
Subject: [Swift-devel] post commit email notify
In-Reply-To: <1473659805.199701287680290433.JavaMail.root@zimbra.anl.gov>
References: <AANLkTiksjQ0JbHpfVdi8XdArj-GW=LLfwYCVVHDZHRBW@mail.gmail.com>
	<1473659805.199701287680290433.JavaMail.root@zimbra.anl.gov>
Message-ID: <AANLkTi=4YpgzPJTZG1u=GJ5oSrhtPfsSk56Ei1PvtoGT@mail.gmail.com>

as far as i know there are no emails being generated for commits at the
moment i was going to look into having it turned on...so, i'm not sure what
the bounces are that you're getting. i don't think we'd need another email
list just for this purpose...i *think* there's a way to set it up (at least
i seem to remember we did this for the sidgrid project) such that we can
have it send a commit 'summary' and we could adjust how frequently that is
sent out--so maybe just once a day or even once a week...i will look into
it, but i just didn't want anyone to feel overwhelmed with spam :)

~sk

On Thu, Oct 21, 2010 at 11:58 AM, Michael Wilde <wilde at mcs.anl.gov> wrote:

> I think Im OK with that. Or, to a new list swift-commits? How is the being
> routed currently? To the Argonne lists?
>
> I think that Im getting some kind of list approval bounces for some of this
> traffic that I'm currently ignoring, and perhaps thats causing a problem.
>
> - Mike
>
>
> ----- "Sarah Kenny" <skenny at uchicago.edu> wrote:
> > would anyone have an objection to having an email notification sent to
> swift-devel when commits are made to trunk? i thought this might be helpful,
> but i'm not sure if others think this might be overkill (?)
> >
> > ~sk
> >
> > _______________________________________________ Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101021/b6aaa3f0/attachment.html>

From wozniak at mcs.anl.gov  Thu Oct 21 12:07:55 2010
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Thu, 21 Oct 2010 12:07:55 -0500 (CDT)
Subject: [Swift-devel] post commit email notify
In-Reply-To: <AANLkTi=4YpgzPJTZG1u=GJ5oSrhtPfsSk56Ei1PvtoGT@mail.gmail.com>
References: <AANLkTiksjQ0JbHpfVdi8XdArj-GW=LLfwYCVVHDZHRBW@mail.gmail.com>
	<1473659805.199701287680290433.JavaMail.root@zimbra.anl.gov>
	<AANLkTi=4YpgzPJTZG1u=GJ5oSrhtPfsSk56Ei1PvtoGT@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1010211206240.1901@wozniak-desktop.mcs.anl.gov>


There is a swift-commit list- I guess you are not currently on that?

On Thu, 21 Oct 2010, Sarah Kenny wrote:

> as far as i know there are no emails being generated for commits at the
> moment i was going to look into having it turned on...so, i'm not sure what
> the bounces are that you're getting. i don't think we'd need another email
> list just for this purpose...i *think* there's a way to set it up (at least
> i seem to remember we did this for the sidgrid project) such that we can
> have it send a commit 'summary' and we could adjust how frequently that is
> sent out--so maybe just once a day or even once a week...i will look into
> it, but i just didn't want anyone to feel overwhelmed with spam :)
>
> ~sk
>
> On Thu, Oct 21, 2010 at 11:58 AM, Michael Wilde <wilde at mcs.anl.gov> wrote:
>
>> I think Im OK with that. Or, to a new list swift-commits? How is the being
>> routed currently? To the Argonne lists?
>>
>> I think that Im getting some kind of list approval bounces for some of this
>> traffic that I'm currently ignoring, and perhaps thats causing a problem.
>>
>> - Mike
>>
>>
>> ----- "Sarah Kenny" <skenny at uchicago.edu> wrote:
>>> would anyone have an objection to having an email notification sent to
>> swift-devel when commits are made to trunk? i thought this might be helpful,
>> but i'm not sure if others think this might be overkill (?)
>>>
>>> ~sk
>>>
>>> _______________________________________________ Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>>
>> --
>> Michael Wilde
>> Computation Institute, University of Chicago
>> Mathematics and Computer Science Division
>> Argonne National Laboratory
>>
>>
>

-- 
Justin M Wozniak


From hategan at mcs.anl.gov  Thu Oct 21 12:07:18 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 21 Oct 2010 10:07:18 -0700
Subject: [Swift-devel] post commit email notify
In-Reply-To: <AANLkTiksjQ0JbHpfVdi8XdArj-GW=LLfwYCVVHDZHRBW@mail.gmail.com>
References: <AANLkTiksjQ0JbHpfVdi8XdArj-GW=LLfwYCVVHDZHRBW@mail.gmail.com>
Message-ID: <1287680838.15993.3.camel@blabla2.none>

If you subscribe to swift-commits at ci, you will get the commit mails.


Mihael

On Thu, 2010-10-21 at 11:43 -0500, Sarah Kenny wrote:
> would anyone have an objection to having an email notification sent to
> swift-devel when commits are made to trunk? i thought this might be
> helpful, but i'm not sure if others think this might be overkill (?)
> 
> ~sk
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel




From skenny at uchicago.edu  Thu Oct 21 12:09:16 2010
From: skenny at uchicago.edu (Sarah Kenny)
Date: Thu, 21 Oct 2010 12:09:16 -0500
Subject: [Swift-devel] post commit email notify
In-Reply-To: <1287680838.15993.3.camel@blabla2.none>
References: <AANLkTiksjQ0JbHpfVdi8XdArj-GW=LLfwYCVVHDZHRBW@mail.gmail.com>
	<1287680838.15993.3.camel@blabla2.none>
Message-ID: <AANLkTi=3vKRDbUGAAYjJt8HOYj_-Rn12VCCphY+SQqNB@mail.gmail.com>

great thanks! will do

On Thu, Oct 21, 2010 at 12:07 PM, Mihael Hategan <hategan at mcs.anl.gov>wrote:

> If you subscribe to swift-commits at ci, you will get the commit mails.
>
>
> Mihael
>
> On Thu, 2010-10-21 at 11:43 -0500, Sarah Kenny wrote:
> > would anyone have an objection to having an email notification sent to
> > swift-devel when commits are made to trunk? i thought this might be
> > helpful, but i'm not sure if others think this might be overkill (?)
> >
> > ~sk
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101021/670c9606/attachment.html>

From wilde at mcs.anl.gov  Thu Oct 21 12:24:38 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 21 Oct 2010 11:24:38 -0600 (GMT-06:00)
Subject: [Swift-devel] Throttling issues for coasters on OSG
Message-ID: <1277288820.200501287681878987.JavaMail.root@zimbra.anl.gov>

Mihael, All, can you provide some guidance on this issue:

Allan is starting coasters "out of band" on OSG, using passive-persistent servers.

He's also setting the throttles for each site manually in sites.xml, somewhat proportional to each site's cpu count.

Not surprisingly (in hindsight) what he's seeing suggests that this approach is not good: it causes Swift to commit jobs to sites where all the running coasters are already busy, while leaving many running coasters on other sites idle.

Allan will now test with the throttles removed to see if the default scheduling algorithm will do a better job. We expect that it will.

But we'd like to explore 2 changes to make the scheduler perform optimally, and see (a) what you think of them and (b) if desirable, how t implement them

Change 1: bias scheduling based on job start rather than job completion. We have discussed this many times in the past, and it seems more optimal in all cases I can imagine. Do you agree? Can you point us to code modules where we can try this and evaluate it?

Change 2: for coasters, use knowledge of how many worker slots are registered on each site to always keep the throttle set to exactly this value, and hence to ensure that, given sufficient ready tasks, all coaster slots are always filled?

I think that for Allan's workload (many hundred thousand 5 minute tasks), change 2 will be most effective and in almost all cases desirable.

A third issue relates to staging, and is perhaps a non-issue if change 2 works. The issue is: if we use provider staging for this workload, which is likely desirable, will data staging be done at the time a worker pulls a job, or will it be done in advance of that, and asynchronously with job start, just as it would be with ordinary staging?

Thanks,

- Mike



From hategan at mcs.anl.gov  Thu Oct 21 13:02:05 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 21 Oct 2010 11:02:05 -0700
Subject: [Swift-devel] Re: Throttling issues for coasters on OSG
In-Reply-To: <1277288820.200501287681878987.JavaMail.root@zimbra.anl.gov>
References: <1277288820.200501287681878987.JavaMail.root@zimbra.anl.gov>
Message-ID: <1287684125.16370.2.camel@blabla2.none>

On Thu, 2010-10-21 at 11:24 -0600, Michael Wilde wrote:

> Change 1: bias scheduling based on job start rather than job
> completion. We have discussed this many times in the past, and it
> seems more optimal in all cases I can imagine. Do you agree? Can you
> point us to code modules where we can try this and evaluate it?
> 
> Change 2: for coasters, use knowledge of how many worker slots are
> registered on each site to always keep the throttle set to exactly
> this value, and hence to ensure that, given sufficient ready tasks,
> all coaster slots are always filled?
> 
> I think that for Allan's workload (many hundred thousand 5 minute
> tasks), change 2 will be most effective and in almost all cases
> desirable.

I dare say not almost, but all cases.

> 
> A third issue relates to staging, and is perhaps a non-issue if change
> 2 works. The issue is: if we use provider staging for this workload,
> which is likely desirable, will data staging be done at the time a
> worker pulls a job, or will it be done in advance of that, and
> asynchronously with job start, just as it would be with ordinary
> staging?

Staging is done after the job is sent to the worker. This is unlike it
would be done with ordinary staging.

Mihael



From aespinosa at cs.uchicago.edu  Thu Oct 21 17:57:49 2010
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Thu, 21 Oct 2010 17:57:49 -0500
Subject: [Swift-devel] the persistence of the persistent coaster service.
Message-ID: <AANLkTi=fi8Urgx6itsg7dDJjS1JK+z9H2Z4NisKP9EG+@mail.gmail.com>

Hi,

When I'm reusing the coaster service onto the next swift session, i
get reply timeouts in the CHANNELCONFIG command:


swift-r3685 cog-r2913

RunID: extract
Progress:
Progress:  uninitialized:2  Finished in previous run:2
Progress:  uninitialized:2  Finished in previous run:2
Progress:  Stage in:99  Submitting:1  Finished in previous run:102
Find: https://communicado.ci.uchicago.edu:61999
Find:  keepalive(120), reconnect - https://communicado.ci.uchicago.edu:61999
Progress:  Stage in:92  Submitting:8  Finished in previous run:102
Passive queue processor initialized. Callback URI is http://128.135.125.17:60999
Progress:  Stage in:71  Submitting:2  Submitted:27  Finished in previous run:102
Progress:  Stage in:29  Submitting:1  Submitted:70  Finished in previous run:102

**Abord** (Ctrl-C)
** rerun/ resume workflow **
swift-r3685 cog-r2913

RunID: extract
Progress:
Progress:  uninitialized:3  Finished in previous run:2
Progress:  Stage in:99  Submitting:1  Finished in previous run:102
Find: https://communicado.ci.uchicago.edu:61999
Find:  keepalive(120), reconnect - https://communicado.ci.uchicago.edu:61999
Progress:  Stage in:92  Submitting:8  Finished in previous run:102
Progress:  Stage in:92  Submitting:8  Finished in previous run:102
Progress:  Stage in:92  Submitting:8  Finished in previous run:102
Progress:  Stage in:92  Submitting:8  Finished in previous run:102
Command(1, CHANNELCONFIG): handling reply timeout;
sendReqTime=101021-174124.460, sendTime=101021-174124.471,
now=101021-174324.492
Command(1, CHANNELCONFIG)fault was: Reply timeout
org.globus.cog.karajan.workflow.service.ReplyTimeoutException
        at org.globus.cog.karajan.workflow.service.commands.Command.handleReplyTimeout(Command.java:280)
        at org.globus.cog.karajan.workflow.service.commands.Command$Timeout.run(Command.java:285)
        at java.util.TimerThread.mainLoop(Timer.java:512)
        at java.util.TimerThread.run(Timer.java:462)
Progress:  Stage in:92  Submitting:7  Submitted:1  Finished in previous run:102

My sites.xml sets the persistent service to work in passive mode.


thanks,
-Allan

-- 
Allan M. Espinosa <http://amespinosa.wordpress.com>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>


From skenny at uchicago.edu  Tue Oct 26 12:52:47 2010
From: skenny at uchicago.edu (Sarah Kenny)
Date: Tue, 26 Oct 2010 12:52:47 -0500
Subject: [Swift-devel] post commit email notify
In-Reply-To: <AANLkTi=3vKRDbUGAAYjJt8HOYj_-Rn12VCCphY+SQqNB@mail.gmail.com>
References: <AANLkTiksjQ0JbHpfVdi8XdArj-GW=LLfwYCVVHDZHRBW@mail.gmail.com>
	<1287680838.15993.3.camel@blabla2.none>
	<AANLkTi=3vKRDbUGAAYjJt8HOYj_-Rn12VCCphY+SQqNB@mail.gmail.com>
Message-ID: <AANLkTi=K2zE4DU8=ERwY_jHfr15mtyDaNoHJzfok+vx7@mail.gmail.com>

i think the list admin needs to approve my subscription (?) i submitted the
form but haven't received a confirmation or any emails...

~sk

On Thu, Oct 21, 2010 at 12:09 PM, Sarah Kenny <skenny at uchicago.edu> wrote:

> great thanks! will do
>
>
> On Thu, Oct 21, 2010 at 12:07 PM, Mihael Hategan <hategan at mcs.anl.gov>wrote:
>
>> If you subscribe to swift-commits at ci, you will get the commit mails.
>>
>>
>> Mihael
>>
>> On Thu, 2010-10-21 at 11:43 -0500, Sarah Kenny wrote:
>> > would anyone have an objection to having an email notification sent to
>> > swift-devel when commits are made to trunk? i thought this might be
>> > helpful, but i'm not sure if others think this might be overkill (?)
>> >
>> > ~sk
>> > _______________________________________________
>> > Swift-devel mailing list
>> > Swift-devel at ci.uchicago.edu
>> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101026/21a49d97/attachment.html>

From aespinosa at cs.uchicago.edu  Tue Oct 26 13:12:33 2010
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Tue, 26 Oct 2010 13:12:33 -0500
Subject: [Swift-devel] Re: Throttling issues for coasters on OSG
In-Reply-To: <1277288820.200501287681878987.JavaMail.root@zimbra.anl.gov>
References: <1277288820.200501287681878987.JavaMail.root@zimbra.anl.gov>
Message-ID: <AANLkTi=4jndQu7yDLqQ=6CzKRay_5YE5Q8OZ9LaGm96X@mail.gmail.com>

Based on the documentation, I should be using "use.provider.staging=proxy"
right?

Thanks,
-Allan

2010/10/21 Michael Wilde <wilde at mcs.anl.gov>:
>
> I think that for Allan's workload (many hundred thousand 5 minute tasks),
> change 2 will be most effective and in almost all cases desirable.
>
> A third issue relates to staging, and is perhaps a non-issue if change 2
> works. The issue is: if we use provider staging for this workload, which is
> likely desirable, will data staging be done at the time a worker pulls a job,
> or will it be done in advance of that, and asynchronously with job start, just
> as it would be with ordinary staging?

-- 
Allan M. Espinosa <http://amespinosa.wordpress.com>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>


From wilde at mcs.anl.gov  Tue Oct 26 17:19:33 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Tue, 26 Oct 2010 16:19:33 -0600 (GMT-06:00)
Subject: [Swift-devel] post commit email notify
In-Reply-To: <AANLkTi=K2zE4DU8=ERwY_jHfr15mtyDaNoHJzfok+vx7@mail.gmail.com>
Message-ID: <623646527.384161288131573797.JavaMail.root@zimbra.anl.gov>

I couldnt find the prior request so I just added you. 

- Mike 

----- "Sarah Kenny" <skenny at uchicago.edu> wrote: 
> i think the list admin needs to approve my subscription (?) i submitted the form but haven't received a confirmation or any emails... 
> 
> ~sk 
> 
> 
> On Thu, Oct 21, 2010 at 12:09 PM, Sarah Kenny < skenny at uchicago.edu > wrote: 
> 

great thanks! will do 


> 
> 
> 
> On Thu, Oct 21, 2010 at 12:07 PM, Mihael Hategan < hategan at mcs.anl.gov > wrote: 
> 

If you subscribe to swift-commits at ci, you will get the commit mails. 
> 
> 
> Mihael 
> 



> On Thu, 2010-10-21 at 11:43 -0500, Sarah Kenny wrote: 
> > would anyone have an objection to having an email notification sent to 
> > swift-devel when commits are made to trunk? i thought this might be 
> > helpful, but i'm not sure if others think this might be overkill (?) 
> > 
> > ~sk 
> > _______________________________________________ 
> 


> Swift-devel mailing list 
> > Swift-devel at ci.uchicago.edu 
> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel 
> 
> 
> 
> 
> 
> _______________________________________________ Swift-devel mailing list Swift-devel at ci.uchicago.edu http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel 

-- 
Michael Wilde 
Computation Institute, University of Chicago 
Mathematics and Computer Science Division 
Argonne National Laboratory 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101026/7ba08f25/attachment.html>

From wilde at mcs.anl.gov  Tue Oct 26 17:36:12 2010
From: wilde at mcs.anl.gov (wilde at mcs.anl.gov)
Date: Tue, 26 Oct 2010 16:36:12 -0600 (GMT-06:00)
Subject: [Swift-devel] post commit email notify
In-Reply-To: <1494837193.384571288132444367.JavaMail.root@zimbra.anl.gov>
Message-ID: <1373748036.384671288132572537.JavaMail.root@zimbra.anl.gov>

In tracking this down, I realized that swift commit messages are sent to swift-commit lists at both ci.uchicago.edu and globus.org. 

To date the globus.org messages were piling up waiting for moderator approval. Ive now deleted these older messages and added noreply at svn.ci.uchicago.edu as a list subscriber to swift-commit at globus.org so these will no longer send moderator requests. 

Mihael and Justin, you are the only users subscribed there; you may get double commit notifications now, untill you unsubscribe from the globus.org list. 

- Mike 


----- "Michael Wilde" <wilde at mcs.anl.gov> wrote: 
> 
> I couldnt find the prior request so I just added you. 
> 
> - Mike 
> 
> ----- "Sarah Kenny" <skenny at uchicago.edu> wrote: 
> > i think the list admin needs to approve my subscription (?) i submitted the form but haven't received a confirmation or any emails... 
> > 
> > ~sk 
> > 
> > 
> > On Thu, Oct 21, 2010 at 12:09 PM, Sarah Kenny < skenny at uchicago.edu > wrote: 
> > 

great thanks! will do 


> > 
> > 
> > 
> > On Thu, Oct 21, 2010 at 12:07 PM, Mihael Hategan < hategan at mcs.anl.gov > wrote: 
> > 

If you subscribe to swift-commits at ci, you will get the commit mails. 
> > 
> > 
> > Mihael 
> > 



> > On Thu, 2010-10-21 at 11:43 -0500, Sarah Kenny wrote: 
> > > would anyone have an objection to having an email notification sent to 
> > > swift-devel when commits are made to trunk? i thought this might be 
> > > helpful, but i'm not sure if others think this might be overkill (?) 
> > > 
> > > ~sk 
> > > _______________________________________________ 
> > 


> Swift-devel mailing list 
> > > Swift-devel at ci.uchicago.edu 
> > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel 
> > 
> > 
> > 
> > 
> > 
> > _______________________________________________ Swift-devel mailing list Swift-devel at ci.uchicago.edu http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel 
> 
> -- 
> Michael Wilde 
> Computation Institute, University of Chicago 
> Mathematics and Computer Science Division 
> Argonne National Laboratory 
> 
> 
> _______________________________________________ Swift-devel mailing list Swift-devel at ci.uchicago.edu http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel 

-- 
Michael Wilde 
Computation Institute, University of Chicago 
Mathematics and Computer Science Division 
Argonne National Laboratory 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101026/48f92e8e/attachment.html>

From hategan at mcs.anl.gov  Tue Oct 26 18:11:51 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 26 Oct 2010 16:11:51 -0700
Subject: [Swift-devel] Re: Throttling issues for coasters on OSG
In-Reply-To: <AANLkTi=4jndQu7yDLqQ=6CzKRay_5YE5Q8OZ9LaGm96X@mail.gmail.com>
References: <1277288820.200501287681878987.JavaMail.root@zimbra.anl.gov>
	<AANLkTi=4jndQu7yDLqQ=6CzKRay_5YE5Q8OZ9LaGm96X@mail.gmail.com>
Message-ID: <1288134711.18201.3.camel@blabla2.none>

No.

You'd use "use.provider.staging=true" and then stagingMethod=proxy in
sites.xml (but that is the default).

Mihael

On Tue, 2010-10-26 at 13:12 -0500, Allan Espinosa wrote:
> Based on the documentation, I should be using "use.provider.staging=proxy"
> right?
> 
> Thanks,
> -Allan
> 
> 2010/10/21 Michael Wilde <wilde at mcs.anl.gov>:
> >
> > I think that for Allan's workload (many hundred thousand 5 minute tasks),
> > change 2 will be most effective and in almost all cases desirable.
> >
> > A third issue relates to staging, and is perhaps a non-issue if change 2
> > works. The issue is: if we use provider staging for this workload, which is
> > likely desirable, will data staging be done at the time a worker pulls a job,
> > or will it be done in advance of that, and asynchronously with job start, just
> > as it would be with ordinary staging?
> 




From wilde at mcs.anl.gov  Wed Oct 27 13:10:40 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 27 Oct 2010 12:10:40 -0600 (GMT-06:00)
Subject: [Swift-devel] minor issue in swift tests - path to ifocnfig
Message-ID: <1448094822.415211288203040540.JavaMail.root@zimbra.anl.gov>

Justin, I get:

bri$ ./nightly.sh 
HTML_OUTPUT: /home/wilde/swift/src/trunk/cog/modules/swift/tests/run-2010-10-27/tests-2010-10-27.html
./nightly.sh: line 764: ifconfig: command not found

I dont have /sbin in my path; maybe others as well? (Seems not to be default on eg bridled at ci)

Maybe fully qualify its usa in the test suite as /sbin/ifconfig?

- Mike


From wozniak at mcs.anl.gov  Wed Oct 27 13:24:17 2010
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Wed, 27 Oct 2010 13:24:17 -0500 (CDT)
Subject: [Swift-devel] Re: minor issue in swift tests - path to ifocnfig
In-Reply-To: <1448094822.415211288203040540.JavaMail.root@zimbra.anl.gov>
References: <1448094822.415211288203040540.JavaMail.root@zimbra.anl.gov>
Message-ID: <alpine.DEB.2.00.1010271324050.29888@wozniak-desktop.mcs.anl.gov>


Thanks for the report, fixed.

On Wed, 27 Oct 2010, Michael Wilde wrote:

> Justin, I get:
>
> bri$ ./nightly.sh
> HTML_OUTPUT: /home/wilde/swift/src/trunk/cog/modules/swift/tests/run-2010-10-27/tests-2010-10-27.html
> ./nightly.sh: line 764: ifconfig: command not found
>
> I dont have /sbin in my path; maybe others as well? (Seems not to be default on eg bridled at ci)
>
> Maybe fully qualify its usa in the test suite as /sbin/ifconfig?
>
> - Mike

-- 
Justin M Wozniak


From wilde at mcs.anl.gov  Wed Oct 27 16:31:54 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 27 Oct 2010 15:31:54 -0600 (GMT-06:00)
Subject: [Swift-devel] Need SGE provider in trunk
Message-ID: <711014488.428141288215114926.JavaMail.root@zimbra.anl.gov>

Mihael, can you merge the latest SGE provider code to trunk from stable?

Glen needs it for production runs.

Thanks,

Mike


From hategan at mcs.anl.gov  Wed Oct 27 19:06:02 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 27 Oct 2010 17:06:02 -0700
Subject: [Swift-devel] Re: Need SGE provider in trunk
In-Reply-To: <711014488.428141288215114926.JavaMail.root@zimbra.anl.gov>
References: <711014488.428141288215114926.JavaMail.root@zimbra.anl.gov>
Message-ID: <1288224362.23313.11.camel@blabla2.none>

I thought I did:

From: 	Mihael Hategan <hategan at mcs.anl.gov>
To: 	Michael Wilde <wilde at mcs.anl.gov>
Cc: 	Swift Devel <swift-devel at ci.uchicago.edu>
Subject: 	Re: [Swift-devel] Re: Need to integrate SGE provider into
trunk
Date: 	Sat, 02 Oct 2010 12:20:10 -0700

I merged the local-scheduler module from the stable branch to trunk in
cog r2893.


On Wed, 2010-10-27 at 15:31 -0600, Michael Wilde wrote:
> Mihael, can you merge the latest SGE provider code to trunk from stable?
> 
> Glen needs it for production runs.
> 
> Thanks,
> 
> Mike




From aespinosa at cs.uchicago.edu  Wed Oct 27 20:06:09 2010
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Wed, 27 Oct 2010 20:06:09 -0500
Subject: [Swift-devel] ext struct mapper on provider staging
Message-ID: <AANLkTi=ZAKZW7BRL14h6haf7cAZevqoAP3OJjXN-9U6u@mail.gmail.com>

Somehow when provider staging is enabled an additional "." is being
added to the path of the data objects which gives invalid path
exceptions in the jobs.  The problem does not occur in swift's normal
usage.

RunID: scratch
Progress:
Progress:  uninitialized:2  Finished in previous run:2
Progress:  uninitialized:2  Finished in previous run:2
Progress:  Selecting site:366  Stage in:1  Active:3  Finished in previous run:2
Execution failed:
        Trying to release previously released contact: 19
Exception in extract:
Arguments: [stat=LGU, extract_sgt=1, slon=-119.06587, slat=34.10819,
rupmodfile=gpfs/pads/swift/aespinosa/science/cybershake/RuptureVariations/113/4/113_4.txt.variation-s0017-h0008,
sgt_xfile=LGU/LGU_fx_664.sgt, sgt_yfile=LGU/LGU_fy_664.sgt,
extract_sgt_xfile=gpfs/pads/swift/aespinosa/science/cybershake/Results/LGU/113/4/LGU_113_4_subfx.sgt,
extract_sgt_yfile=gpfs/pads/swift/aespinosa/science/cybershake/Results/LGU/113/4/LGU_113_4_subfy.sgt]
Host: PADS
Directory: postproc-scratch/jobs/g/extract-gg6rfr0kTODO: outs
----

Caused by:
        org.griphyn.vdl.mapping.InvalidPathException: Invalid path
(..y) for org.griphyn.vdl.mapping.RootDataNode identifier
tag:benc at ci.uchicago.edu,2008:swift:dataset:20101027-2001-iexur5ud:720000026179
type Sgt with no value at dataset=sub (not closed)

log entry:
2010-10-27 20:02:07,534-0500 INFO  ExecutionContext Detailed exception:
Exception in extract:
Arguments: [stat=LGU, extract_sgt=1, slon=-119.06587, slat=34.10819,
rupmodfile=gpfs/pads/swift/aespinosa/science/cybe
rshake/RuptureVariations/113/4/113_4.txt.variation-s0017-h0008,
sgt_xfile=LGU/LGU_fx_664.sgt, sgt_yfile=LGU/LGU_fy_664
.sgt, extract_sgt_xfile=gpfs/pads/swift/aespinosa/science/cybershake/Results/LGU/113/4/LGU_113_4_subfx.sgt,
extract_sg
t_yfile=gpfs/pads/swift/aespinosa/science/cybershake/Results/LGU/113/4/LGU_113_4_subfy.sgt]
Host: PADS
Directory: postproc-scratch/jobs/g/extract-gg6rfr0kTODO: outs
----

Caused by: org.griphyn.vdl.mapping.InvalidPathException: Invalid path
(..y) for org.griphyn.vdl.mapping.RootDataNode i
dentifier tag:benc at ci.uchicago.edu,2008:swift:dataset:20101027-2001-iexur5ud:720000026179
type Sgt with no value at da
taset=sub (not closed)
Caused by: org.griphyn.vdl.mapping.InvalidPathException: Invalid path
(..y) for org.griphyn.vdl.mapping.RootDataNode i
dentifier tag:benc at ci.uchicago.edu,2008:swift:dataset:20101027-2001-iexur5ud:720000026179
type Sgt with no value at da
taset=sub (not closed)
        at org.globus.cog.karajan.workflow.nodes.functions.KException.function(KException.java:29)
        at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:27)
        at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWith
Arguments.java:192)
        at org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
        at org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
        at org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
        at org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
        at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
        at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
        at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28)
        at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
        at org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:26)
        at org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63)
        at org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:238)
        at org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:289)
        at org.globus.cog.karajan.workflow.nodes.FlowNode.controlEvent(FlowNode.java:402)
        at org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:343)
        at org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
        at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:44)
        at edu.emory.mathcs.backport.java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:431)
        at edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:643)
        at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:668)
        at java.lang.Thread.run(Thread.java:619)
Caused by: org.griphyn.vdl.mapping.InvalidPathException: Invalid path
(..y) for org.griphyn.vdl.mapping.RootDataNode identifier
tag:benc at ci.uchicago.edu,2008:swift:dataset:20101027-2001-iexur5ud:720000026179
type Sgt with no value at dataset=sub (not closed)
Caused by: org.griphyn.vdl.mapping.InvalidPathException: Invalid path
(..y) for org.griphyn.vdl.mapping.RootDataNode identifier
tag:benc at ci.uchicago.edu,2008:swift:dataset:20101027-2001-iexur5ud:720000026179
type Sgt with no value at dataset=sub (not closed)
        at org.griphyn.vdl.karajan.lib.AppStageouts.post(AppStageouts.java:56)
        ... 22 more
Caused by: org.griphyn.vdl.mapping.InvalidPathException: Invalid path
(..y) for org.griphyn.vdl.mapping.RootDataNode identifier
tag:benc at ci.uchicago.edu,2008:swift:dataset:20101027-2001-iexur5ud:720000026179
type Sgt with no value at dataset=sub (not closed)
        at org.griphyn.vdl.mapping.AbstractDataNode.getField(AbstractDataNode.java:179)
        at org.griphyn.vdl.karajan.lib.AppStageouts.post(AppStageouts.java:40)
        ... 22 more
2010-10-27 20:02:07,538-0500 DEBUG AbstractDataNode Notifying listener
"F/org.griphyn.vdl.mapping.DataNode identifier
tag:benc at ci.uchicago.edu,2008:swift:dataset:20101027-2001-iexur5ud:720000028126
type Variation with no value at dataset=vars path=[337] (closed)"
about "org.griphyn.vdl.mapping.DataNode identifier
tag:benc at ci.uchicago.edu,2008:swift:dataset:20101027-2001-iexur5ud:720000028126
type Variation with no value at dataset=vars path=[337] (closed)"
2010-10-27 20:02:07,506-0500 DEBUG AbstractDataNode closed
org.griphyn.vdl.mapping.RootDataNode identifier
tag:benc at ci.uchicago.edu,2008:swift:dataset:20101027-2001-iexur5ud:720000029854
type VariationFile with no value at dataset=file (closed)


-- 
Allan M. Espinosa <http://amespinosa.wordpress.com>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>


From wilde at mcs.anl.gov  Wed Oct 27 22:21:06 2010
From: wilde at mcs.anl.gov (wilde at mcs.anl.gov)
Date: Wed, 27 Oct 2010 21:21:06 -0600 (GMT-06:00)
Subject: [Swift-devel] Re: Need SGE provider in trunk
In-Reply-To: <957321276.438161288235938411.JavaMail.root@zimbra.anl.gov>
Message-ID: <463124245.438261288236066221.JavaMail.root@zimbra.anl.gov>


----- "Mihael Hategan" <hategan at mcs.anl.gov> wrote:

> I thought I did:

Ah, right, thanks for the reminder!  I remember now that I thanked you for that :(

I'll follow up with Glen after he updates his trunk (per his response to your message).

- Mike

> 
> From: 	Mihael Hategan <hategan at mcs.anl.gov>
> To: 	Michael Wilde <wilde at mcs.anl.gov>
> Cc: 	Swift Devel <swift-devel at ci.uchicago.edu>
> Subject: 	Re: [Swift-devel] Re: Need to integrate SGE provider into
> trunk
> Date: 	Sat, 02 Oct 2010 12:20:10 -0700
> 
> I merged the local-scheduler module from the stable branch to trunk
> in
> cog r2893.
> 
> 
> On Wed, 2010-10-27 at 15:31 -0600, Michael Wilde wrote:
> > Mihael, can you merge the latest SGE provider code to trunk from
> stable?
> > 
> > Glen needs it for production runs.
> > 
> > Thanks,
> > 
> > Mike

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From hategan at mcs.anl.gov  Thu Oct 28 03:05:32 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 28 Oct 2010 01:05:32 -0700
Subject: [Swift-devel] Re: Need SGE provider in trunk
In-Reply-To: <463124245.438261288236066221.JavaMail.root@zimbra.anl.gov>
References: <463124245.438261288236066221.JavaMail.root@zimbra.anl.gov>
Message-ID: <1288253132.25137.3.camel@blabla2.none>

On Wed, 2010-10-27 at 21:21 -0600, wilde at mcs.anl.gov wrote:
> ----- "Mihael Hategan" <hategan at mcs.anl.gov> wrote:
> 
> > I thought I did:
> 
> Ah, right, thanks for the reminder!  I remember now that I thanked you for that :(

It happens. Now where was I going before I started writing this email?

> 
> I'll follow up with Glen after he updates his trunk (per his response to your message).

It may, at times, prove useful (in the event that there are no major API
changes) to simply drop a provider jar file into the lib directory
(while also removing the old one). I can't guarantee this will work
between versions, but it may be worth a try.

Mihael



From wilde at mcs.anl.gov  Thu Oct 28 15:45:19 2010
From: wilde at mcs.anl.gov (wilde at mcs.anl.gov)
Date: Thu, 28 Oct 2010 14:45:19 -0600 (GMT-06:00)
Subject: [Swift-devel] Use of ControlMaster ssh channels
In-Reply-To: <111612260.476851288296967405.JavaMail.root@zimbra.anl.gov>
Message-ID: <2020199357.479271288298719548.JavaMail.root@zimbra.anl.gov>

Mihael,

Last year, Narayan and the JGI folks were experimenting with ssh control master channels to achieve password-free logins to remote machines.

You said at the time that the ssh classes that Swift uses would not be able to use this mechanism. And indeed, I just tried it and have so far not been able to get Swift to avoid the password/passphrase prompt, which confirms your assessment.

But, do you know if this is conceptually possible? Maybe with some enhanced ssh classes?

What ssh package are you using currently in cog?

Thanks,

Mike




From hategan at mcs.anl.gov  Thu Oct 28 16:07:14 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 28 Oct 2010 14:07:14 -0700
Subject: [Swift-devel] Re: Use of ControlMaster ssh channels
In-Reply-To: <2020199357.479271288298719548.JavaMail.root@zimbra.anl.gov>
References: <2020199357.479271288298719548.JavaMail.root@zimbra.anl.gov>
Message-ID: <1288300034.26601.8.camel@blabla2.none>

On Thu, 2010-10-28 at 14:45 -0600, wilde at mcs.anl.gov wrote:
> Mihael,
> 
> Last year, Narayan and the JGI folks were experimenting with ssh
> control master channels to achieve password-free logins to remote
> machines.
> 
> You said at the time that the ssh classes that Swift uses would not be
> able to use this mechanism. And indeed, I just tried it and have so
> far not been able to get Swift to avoid the password/passphrase
> prompt, which confirms your assessment.
> 
> But, do you know if this is conceptually possible? Maybe with some
> enhanced ssh classes?

The problem here is Java's inability to use Unix domain sockets, which,
as far as I remember, is the mechanism used by the SSH master channels
(as well as the SSH agent).

The essential problem is that of having a connection that survives
outside of a given process (i.e. multiple swift invocations implies in
general multiple jvm invocations) combined with the need for a secure
mechanism to use that connection (plain TCP socket's won't do because
one cannot enforce UNIX permissions on clients).

However, should it be that the JVM is persisted across multiple swift
invocations, the SSH provider already uses a similar mechanism (i.e.
connection is kept and multiple SSH channels are started on that
connection).

> 
> What ssh package are you using currently in cog?

j2ssh.

Mihael



From dk0966 at cs.ship.edu  Sat Oct 30 17:10:53 2010
From: dk0966 at cs.ship.edu (David Kelly)
Date: Sat, 30 Oct 2010 18:10:53 -0400
Subject: [Swift-devel] Swift shell script update
Message-ID: <AANLkTikY39GS+Sm2aLNAS7-0R8OgYgyefGcPY0FXp7Aw@mail.gmail.com>

Hello,

I made an update tonight to the swift shell script related to usage
tracking. The new version replaces netcat with bash's built-in
networking interface. I've tested it on a variety of different
machines and setups, but please let me know if anyone runs into any
issues.

Thanks,
David