runtime: mechanism for monitoring heap size

[bradfitz]

Tracking bug for some way for applications to monitor memory usage, apply backpressure, stay within limits, etc.

Related previous issues: #5049, #14162

[rgooch]

Can you please expand on what you have in mind?

[bradfitz]

I have nothing specific in mind. This bug was filed as part of a triage meeting with a bunch of us. One bug (#5049) was ancient with no activity and one bug (#14162) proposed a solution instead of discussing the problem.

This bug is recognition that there is a problem, and we've heard problem statements and potential solutions (and sometimes both) from a number of people.

The reality is that there are always memory limits, and it'd be nice for the Go runtime to help applications stay within them, through perhaps some combination of limiting itself, and/or helping the application apply backpressure when resources are getting tight. That might involve new runtime API surface to help applications know when things are getting tight.

/cc @nictuku also.

[bradfitz]

Btw, there was lots of good conversation at #14162 and it wasn't our intention to kill it or devalue it. It just didn't fit the proposal process, and we also didn't want to decline it, nor close it as a dup of #5049.

Changing the language is out of scope, so all discussions of things like catching memory allocation failures, language additions like "trymake" or "tryappend", etc, are all not going to happen.

But we can add runtime APIs to help out. That's what this bug is tracking.

/cc @matloob @aclements

[rgooch]

Agreed. "try*" isn't practical. It would require changing too make call-sites and even then would not catch all allocations. Adding runtime.SetSoftMemoryLimit() still seems like the best approach.

[nictuku]

It would be nice to have the ability to set a limit to the memory usage.

After a limit is set, perhaps the runtime could provide a clear indication that we're under memory pressure and that the application should avoid creating new allocations. Example new runtime APIs that would help:

• func InMemoryPushback() bool; or
• func RegisterPushbackFunc(func(inPushback bool))

That would provide a clear signal to the application. How exactly that's decided should be an internal implementation decision and not part of the API. An example implementation, to illustrate: if we limit ourselves to the heap size specified by the user, we could trigger GC whenever the used heap is close to the limit. Then we could enter pushback whenever the GC performance (latency or CPU overhead) is outside certain bounds. Apply smoothing as needed.

The approach suggested by this API has limitations.

For example, it's still possible for an application that is behaving well to do one monstrous allocation after it has checked for the pushback state. This would be common for HTTP and RPC servers that do admittance control at the beginning of the request processing. If the monstrous allocation would bring the memory heap above the limit, Go should probably panic. Since we don't want to change the language to add memory allocation error checks, I think this is fine. And we have no other option :).

Another problem is that deciding what is the right time to pushback can be hard. Whatever the runtime implements, some folks may find it too aggressive (pushing back too much, leading to poor resource utilization) or too conservative (pushing back too late, leading to high latency due to excessive GC). I guess the go team could provide a knob similar to GOGC to control the pushbackiness of the runtime, if folks are really paranoid about it.

[RLH]

The runtime could set up a channel and send a message whenever it completes
a GC. The application could have a heap monitor goroutine (HMG) watching
that channel. Whenever the HMG gets a message it inspects the state of the
heap. To determine the size of the heap the HMG would look at the live heap
size and GOGC. If need be it could adjust GOGC so that the total heap does
not exceed whatever limit the application finds appropriate. If things are
going badly for the application the HMG can start applying back pressure to
whatever part of the application is causing the increase in heap size. The
HMG would be part of the application so a wide variety of application
specific strategies could be implemented.

Trying to pick up the pieces after a failure does not seem doable. Likewise
deciding what is "close to a failure" is very application specific and a
global metric that potentially involves external OS issues such as
co-tenancy as well as other issue well beyond the scope of the Go runtime.
Decisions and actions need to be made well ahead if one expects them to
reliable prevent an OOM.

I believe this is where we were headed in #14162
#14162 and this is a recap of some of
that discussion.

I would be interested in what useful policy could not be implemented using
the HMG mechanism and current runtime mechanisms.

On Tue, Aug 23, 2016 at 1:43 AM, Yves Junqueira notifications@github.com
wrote:

For Google's internal needs, it would be nice to have the ability to set a
limit to the memory usage.

After a limit is set, perhaps the runtime could provide a clear indication
that we're under memory pressure and that the application should avoid
creating new allocations. Example new runtime APIs that would help:

• func InMemoryPushback() bool; or
• func RegisterPushbackFunc(func(inPushback bool))

That would provide a clear signal to the application. How exactly
that's decided should be an internal implementation decision and not part
of the API. An example implementation, to illustrate: if we limit ourselves
to the heap size specified by the user, we could trigger GC whenever the
used heap is close to the limit. Then we could enter pushback whenever the
GC performance (latency or CPU overhead) is outside certain bounds. Apply
smoothing as needed.

The approach suggested by this API has limitations.

For example, it's still possible for an application that is behaving well
to do one monstrous allocation after it has checked for the pushback state.
This would be common for HTTP and RPC servers that do admittance control at
the beginning of the request processing. If the monstrous allocation would
bring the memory heap above the limit, Go should probably panic. Since we
don't want to change the language to add memory allocation error checks, I
think this is fine. And we have no other option :).

Another problem is that deciding what is the right time to pushback can be
hard. Whatever the runtime implements, some folks may find it too
aggressive (pushing back too much, leading to poor resource utilization) or
too conservative (pushing back too late, leading to high latency due to
excessive GC). I guess the go team could provide a knob similar to GOGC to
control the pushbackiness of the runtime, if folks are really paranoid
about it.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#16843 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AA7Wn-x0kWzbQY0w2nI8daJSWBbIHPWHks5qiohsgaJpZM4Jqa25
.

[rgooch]

I previously gave the reasoning why using a channel or a callback to receive memory exceeded events won't work: #14162
That same reasoning applies to a channel whenever a GC run is completed.

To robustly handling exceeding a memory limit the check for the limit has to be part of the allocator, not done after a GC run. This is because you can't afford to wait. If you wait for the next GC run, it may be too late. Consider a single large slice allocation that would put you over the soft limit and would exceed the hard memory limit. You'll get an OOM panic. The same applies to a callback function.

You need to immediately stop the code which is doing the heavy allocating. To do that you need a check in the allocator and you need to send a panic(). It's up to the application to set the soft memory limit at which these optional, catchable panics are sent.

Please, before rehashing old suggestions or coming up with new variants, read through #14162 where I gave the reasoning why a panic and a check in the allocator is needed. Otherwise we keep covering the same old ground.

[quentinmit]

@rgooch If you are allocating giant arrays, you probably know exactly where in your code that is happening, and you can add code there to first check if there is enough memory available. You can even do that using the GC information we're discussing passing down a channel.

I do think there is a race here, but in the opposite case - if code is sitting in a tight loop making many small allocations, your channel read/callback might not run in time to actually trigger a new GC soon enough without OOMing.

[rgooch]

I discussed all this in #14162: you can be reading GOB-encoded data from a network connection. No way to know ahead of time how big it's going to be. Or it can be some other library you don't control where a lot of data are allocated, whether a single huge slice or a lot of small allocations. The point is, you don't know how much will be allocated before you enter the library code and you've got no way to reach in there and stop things if you hit some pre-defined limit. And, as you say, if you're in a loop watching allocations, even if you could stop things, you may not get there in time. Spinning in a loop watching the memory level is grossly expensive. This needs to be tied to the allocator.

[RLH]

This does not propose a callback or channel for delivering a memory
exceeded message or a memory almost exceeded message. At that point it is
already too late. This proposes a mechanism for providing the application
timely information that it can use to avoid the OOM. The application knows
how best to predict memory usage and, if need be, throttle its memory usage.

One suggestion was
func runtime.ReserveOOMBuffer(size uint64)

The application's heap monitor goroutine, HMG, could initially allocate a
large object of the required size and retain a single reference to it. If
the HMG using information provided by the runtime determines that the
current GOGC and live heap size will not support the application's
predicted allocations then it can release that single reference confident
that the next GC will recover those spans and make them available. It the
HMG wants the GC to happen sooner than currently scheduled then it can
lower GOGC using SetGCPercent.

If ReserveOOMBuffer is the API that some Go application needs then this
provides it. The intent of this proposal is to provide the application with
the information it needs to create the abstractions that best fits its need
while minimizing Go's runtime API surface.

On Tue, Aug 23, 2016 at 11:13 AM, rgooch notifications@github.com wrote:

I previously gave the reasoning why using a channel or a callback to
receive memory exceeded events won't work: #14162
#14162
That same reasoning applies to a channel whenever a GC run is completed.

To robustly handling exceeding a memory limit the check for the limit has
to be part of the allocator, not done after a GC run. This is because you
can't afford to wait. If you wait for the next GC run, it may be too late.
Consider a single large slice allocation that would put you over the soft
limit and would exceed the hard memory limit. You'll get an OOM panic.
The same applies to a callback function.

You need to immediately stop the code which is doing the heavy allocating.
To do that you need a check in the allocator and you need to send a
panic(). It's up to the application to set the soft memory limit at which
these optional, catchable panics are sent.

Please, before rehashing old suggestions or coming up with new variants,
read through #14162 #14162 where I
gave the reasoning why a panic and a check in the allocator is needed.
Otherwise we keep covering the same old ground.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#16843 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AA7Wn4rwLDnFazl8ko7MEgqGqjlHlYJKks5qiw4rgaJpZM4Jqa25
.

[dr2chase]

As I read this, #14162 describes a workload where (analogy follows) sometimes the python attempts to swallow a rhino, and if the attempt is not halted ASAP it is guaranteed to end badly. Is it in fact the case that the rhino will never be successfully swallowed? (I can imagine DOS attacks on servers where this might be the case.)

I think that the periodic notification scheme is intended to deal with a python diet of a large number of smaller prey; if an application has the two constraints of m=memory < M and l=latency < L, and if m is affine in workload W (reasonable assumption) and l is also affine in workload W (semi-reasonable), then simply comparing observed m with limit M and observed l with limit L tells you how much more work can be admitted (W' = W * min(M/m, L/l)), with the usual handwaving around unlucky variations in the input and lag in the measurement. It's possible to adjust GOGC up or down if M/m and L/l are substantially different, so as to maximize the workload within constraints -- this however also requires knowledge of the actual GC overhead imposed on the actual application (supposed to be 25% during GC, but high allocation rates change this). One characteristic of this approach is that a newly started application might not snap online immediately at full load, but would increase its intake as it figured out what load it could handle.

But this is no help for intermittent rhino-swallowing.

[jessfraz]

@bradfitz would you be open to me taking some of the ideas from #14162 and applying the Go proposal process so it is considered? As long of course the proposed solution doesn't break the API or change the language.