[Swift-devel] Re: Persistent coaster service fails after several runs

[Mihael Hategan]

I made a change in r2948.

Due to the fact that the array sizes were 32768 (which is exactly the
I/O buffer size that the worker uses) I suspect that they held file
data.

The problem was that the channel needed to hold a persistent buffer
because it wasn't necessary that all of the data in one chunk could be
read at once from the stream/socket. But that persistence was only
needed until the the whole chunk was read, after which it could be
released.

In the BG/P case, I suspect the problem was a lot worse than in Mike's
case because you probably used many more workers.

So please give that a try. There will, of course, always be byte arrays
with the coaster I/O, but they should be GCed eventually.

Mihael

On Tue, 2010-11-30 at 16:57 -0600, Justin M Wozniak wrote:
> I'm on Intrepid so it's an IBM heap dump.  There's a good one there in 
> ~wozniak/Public/heapdumps .
> 
> The byte[]s are definitely associated with TCPChannel but that's all I 
> have been able to figure out so far- I don't see where they are retained.
> 
> It is possible that the reader is generating the bytes faster than the 
> network can push them out, so we just need to tighten up the throttle?
> 
> On Tue, 30 Nov 2010, Mihael Hategan wrote:
> 
> > Can you make a heap dump of the relevant issue?
> >
> > On Tue, 2010-11-30 at 09:59 -0600, Justin M Wozniak wrote:
> >> Along these lines, I'm looking at memory usage in Coasters.  There's a
> >> plot attached below- usage spikes when the workers start running.
> >>
> >> 96% of the usage is byte[] which makes me think it could be KarajanChannel
> >> stuff...
> >>
> >> http://www.ci.uchicago.edu/wiki/bin/view/SWFT/PerformanceNotes#Memory
> >>
> >> On Sun, 28 Nov 2010, Michael Wilde wrote:
> >>
> >>> This fix looks great so far - Ive tested with varying workflow sizes and delays, and have not seen any problems.
> >>>
> >>> - Mike
> >>
> >
> >
> >
>