[Swift-devel] [Bug 253] New: Enhance coaster timeout processing in passive and persistent modes

Sun Feb 6 13:32:35 CST 2011

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=253

           Summary: Enhance coaster timeout processing in passive and
                    persistent modes
           Product: Swift
           Version: unspecified
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: enhancement
          Priority: P1
         Component: SwiftScript language
        AssignedTo: hategan at mcs.anl.gov
        ReportedBy: wilde at mcs.anl.gov


There is concern/evidence that coaster timeout processing is still not working
well in at least passive persistent mode, and possibly ordinary persistent
mode.

The following IM thread between Mihael and Mike on Jan 17 5:05 PM CST describe
the issues and initial diagnosis.

---

6:06:26 PM Michael Wilde: thing 2: the coaster timeout ignore feature
6:06:49 PM Michael Wilde: thing 3: make thing 2 turn of timing for passive as
well, *or* based on a property
6:07:05 PM Michael Wilde: (all those are for SwiftR)
6:07:16 PM Michael Wilde: or, integrate fast back into trunk?
6:07:34 PM Michael Wilde: and then its just thing 3 (out of 2 ; )
6:07:58 PM Mihael: what's 2?
6:08:50 PM Michael Wilde: the feature you added to disable worker timeouts when
using persisten-coasters
6:09:22 PM Michael Wilde: Turns out SwiftR needs to disable those timers, but
its using only passive but not persistent
6:09:44 PM Michael Wilde: so I needed to manually set the ignioreTimeouts flag
in CoasterService
6:11:30 PM Mihael: it's disabled in all cases
6:12:07 PM Michael Wilde: no, not as far as my tests show
6:12:18 PM Michael Wilde: Im using just passive coasters,
6:12:35 PM Michael Wilde: and the workers die after a few minutes idle time
6:12:45 PM Mihael: that code is in the worker
6:12:46 PM Mihael: it got removed
6:12:56 PM Michael Wilde: the flag is set in the persisten-coaster service
startup only, as far as i can tell
6:13:05 PM Michael Wilde: no, im talking about the java side
6:13:22 PM Michael Wilde: the service its telling the worker to shutdown
6:13:25 PM Michael Wilde: after a few minutes
6:13:57 PM Michael Wilde: if i force on the flag that is set on
persisten-coaster startup, then the workers dont time out 
6:14:46 PM Mihael: shouldn't happen
6:14:47 PM Mihael: but I can check
6:14:57 PM Michael Wilde: k, thanks
6:15:10 PM Michael Wilde: so your intent was to disable *all* worker timeout?
6:15:34 PM Michael Wilde: leaving it on for normal auto coasters and off if
anything is running passive or persistent seems to make sense
6:15:37 PM Michael Wilde: i think
6:16:26 PM Mihael: yes
6:16:29 PM Mihael: all worker timeout is disabled
6:16:44 PM Mihael: the workers still die when no service is present due to lack
of heartbeat
6:16:46 PM Michael Wilde: thats not the behavior I am getting
6:16:50 PM Michael Wilde: ah
6:17:17 PM Michael Wilde: but no - when I logged them, they were getting a
shutdown from the service
6:17:29 PM Michael Wilde: would that in *turn* be from lack of heartbeat?
6:18:25 PM Michael Wilde: This is the change I had to make to prevent the
workers from quitting:
6:18:25 PM Mihael: no
6:18:26 PM Michael Wilde:      private synchronized void checkIdleTime() {
+        return;
+        /*
         if (ignoreIdleTime) {
             return;
         }
@@ -238,6 +240,7 @@
                 shutdown();
             }
         }
+        */
     }

     public synchronized void suspend() {
@@ -342,7 +345,8 @@
     }

     public boolean getIgnoreIdleTime() {
-        return ignoreIdleTime;
+        return true; // ignoreIdleTime; MW: set this based on a
swift.properties property or ???
+                     // need to disable idle timeout for the R passive coaster
config
     }

6:18:27 PM Mihael: probably not
6:18:56 PM Michael Wilde: This *seems* to work for me, although "seems" can be
deceiveing 
6:19:00 PM Mihael: the persitent worker manager should not shut down workers
6:19:04 PM Mihael: there may be a bug there
6:19:22 PM Michael Wilde: but thats the point, I am not *running* the
persistent mgr
6:19:36 PM Michael Wilde: Im using normal coaster provider, in passive mode
6:20:05 PM Michael Wilde: so that ignoreIdleTimeout flag is not getting set
6:20:47 PM Michael Wilde: btw my change above to getIgnoreIdleTime doesnt work,
that is not used it seems.
6:21:40 PM Mihael: ah, ok
6:21:44 PM Mihael: right
6:21:48 PM Mihael: I meant passive mode
6:22:00 PM Mihael: "passive worker manager"
6:22:27 PM Mihael: @Override
    protected void removeIdleBlocks() {
        // no removing of idle blocks here

6:22:45 PM Mihael: though they might get shut down by something else
6:22:50 PM Mihael: I will have to check
6:23:03 PM Michael Wilde: Ah, ok. so yes, that may be a bug, and it seems to
get de-bugged if force the immediate return form checkIdleTime.
6:23:07 PM Michael Wilde: k.
6:23:46 PM Michael Wilde: So, back t the fast branch issue:  whats your
judgement on that?  Integrate it into trunk, or integrate selected changes into
fast?
6:24:20 PM Michael Wilde: I was thinking to re-do my small tasks-per-sec tests
between fast and trunk, and see how much fast helps
6:25:01 PM Michael Wilde: I can probably do the needed changes in a local copy
of fast, unless integration of fast is more imminent
6:25:27 PM Mihael: we'll put that into trunk next

-- 
Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching the reporter.

