[Swift-devel] Tracking swift heap overflow culprit

[Mihael Hategan]

On Thu, 2014-01-30 at 19:02 -0600, Michael Wilde wrote:
> (Moving this to my MCS email addr and to swift-devel)
> 
> Mihael, what we are trying to do here is not (initially) change anything in
> Swift memory usage.

I understand. I, however, have tried and I believe it to be a worthwhile
task to continuously identify places where things can be improved.

> 
> We just want to understand the costs in memory of normal Swift operations,
> eg, call a function with N args and M returns; map a file; create an array
> of 1000 1MB strings; etc.

Right. I understand. I think the best way of doing that is to trace
memory use for different sizes of the problem and try to fit a line to
the results. Essentially we need a model there. We can start with models
for simple things, such as the ones you describe. It's possible that
these can be combined to get estimates for more complex problems, but I
am not entirely sure.

> 
> Then, for any program execution, we want to be able to trace - at some
> useful level of granularity - the consumption of Java memory caused by
> these normal Swift activities.
> 
> For example, if a user writes a function that is going to create - and hold
> - 10MB of memory, due to its local variables, then having 10,000 of those
> active at once would consume - and hold - 100GB of RAM.
> 
> My suspicion is that this is exactly what e.g. Sheri's code is doing.  Any
> I further suspect that once we identify what procedures are using most of
> the memory in what way, then we can tune the user code to use much less
> memory,

Maybe. In my experience profiling swift memory usage over the years is
that there are many many things that contribute just a little, but it
adds up.

> 
> We can - by experiment - develop a cost table for common Swift operations.
> But Sheri's code is the most complex Swift scripts that exist. Each has a
> few K lines of swift code.  WIthout some auomated memory usage stats that
> correlate mem consumption to source code, it will be hard to find the
> culprits.

I would love something like that. At the expense of sounding
pessimistic, I also think that it is a hard problem. I think a good
first step in developing such a tool would be to look at existing memory
profilers and learn a bit from them.

> 
> So the question is not how to make Swift use less memor (although thats
> always desirable), but rather first just to create the tools to know how
> much a give program run uses for what.
> 
> Can you suggest affordable ways to get this info?

I believe that some of this can be done with the tracing in the log (if
tracing is enabled). It can be used to count how many times a procedure
is started, what variables are being allocated in that procedure, and so
on. It won't give us information on how much memory each variable takes
(that might be tricky without some jvm-level profiling tools), but it
may be a start.

Mihael