Proposal: Memory allocation measurement on traces to detect bloat/leaks

[johnnyshields]

VM interpreted languages like Ruby, Python, etc. have a heap memory allocator / garbage collector (GC) which can be subject to memory bloat (spikes where lots of memory is suddenly allocated) and memory leaks (gradual accumulation of non-GC'ed objects over time. These can lead to critical problems such as paging-related slowness (primarily a symptom of with bloat) and out-of-memory process kills (primarily a symptom of with leaks).

This blog post has good explanation from Ruby perspective.

OTEL does a great job in analyzing latency/queries/rendering, however, that's often only part of the story especially in complex apps. It would be very useful to be able to look at object memory allocations on each app endpoint.

Here's an example of what Scout APM can do for Ruby/Rails to measure memory. Note that this measurement is done on each trace:

And here's the dramatic result of implementing a single fix on one endpoint discovered using this tool:

Any interest to make a spec for memory related concerns?

[tsloughter]

In Erlang we report memory allocator and GC stats as metrics. What I think may be good, but could be too hard to generalize, is semantic conventions for these metrics to help backends understand every VM's metrics properly and give insight to the user.

[johnnyshields]

I don't think it's too hard to generalize. There can be a set of language-agnostic conventions that may not be applied 100% uniformly across all languages, but this is already the reality of OTEL today. The concept of "memory allocation" as measured in "objects" or "bytes" for example is pretty universal across languages, those are really the two main metrics to target here.

[jsuereth]

Being able to tie allocations directly to a Trace span (or request flow) is highly architectural dependent. I'm not just suggesting it's a cross-language issue, but for many languages it's likely also a web-framework issue.

I do think the ability to create this feature would be awesome, but let's not assume it'll be easy to get coverage across all otel languages. I suspect it'll be downright impossible outside of a few scenarios. So if we wanted to do this, we absolutely MUST NOT make it a requirement.

However, I'd rather see a convention around this and the ability to track at this level being a "feature" of a good web/service framework.

[johnnyshields]

Yes, this would not be a requirement.

[tigrannajaryan]

Not clear if this is a spec issue or a project proposal. Needs more info.

[dyladan]

@open-telemetry/profiling-maintainers I think this might be either a profiling question or a semconv request to add profiling-adjacent attributes. Please take a look at this and let us know what you think.

[felixge]

The profiling SIG is working on a spec that allows to capture different types of samples, including allocation samples. We're also planning on having a way to correlate individual samples with trace and span ids. So the protocol should be able to support the use case outlined here.

But yeah, SDK support is another matter and will depend on the language.