proposal: make it possible to catch failed memory allocations

[rgooch]

There's currently no way to safely allocate memory in Go without running the risk of the process panicing. This makes it very difficult to have an application that automatically scales to fit the machine (or container) resources. Further, for an application to determine how much memory is available for allocation from the OS is non-portable and unreliable, so it's not like an application can wrap allocations inside a function that checks to see if they will fit. The guess it can make about available memory can be under or over the real value, leading to wasted resources or OOM panics.

I realise that running defer() code when a memory allocation has failed could in turn fail, since the recovery code will also need to allocation memory. However, the common case is probably that a large allocation failed, and there is room for small allocations needed for cleanup, so just allowing applications to catch OOM panics would probably help most of the time. If there is another memory allocation failure during recovery, kill the application.

A more complete solution would be to reserve a chunk of memory that is freed/made available during OOM recovery, which is re-reserved once recovery is complete. That would also allow effective recovery if a small memory allocation failed. As long as the recovery code uses less memory than the reserved size, this approach should be robust. The memory would be allocated with the normal internal mechanisms, so it wouldn't be "special". The OOM handling code would be the only place where a reference was kept.

This reservation could be done implicitly, but it's probably cleaner to require the application to enable this feature. That would ensure that applications that don't need OOM avoidance don't have to pay the price of the reserved memory. I suggest the following API:
func runtime.ReserveOOMBuffer(size uint64)

The OOM handler would drop the reference to the reserved memory, call runtime.GC() and then generate a normal panic() rather than panic+kill. The recovery code in the application would be expected to call runtime.ReserveOOMBuffer() again (probably after calling runtime.GC()).

A less attractive option is to add trymake() and tryappend() built-in functions, which return a value,error tuple and a tryinsert() built-in which inserts into a map and returns an error value. I like this less because there are many other ways in which memory is allocated, so one cannot catch them all. It also requires changing a large number of callsites. Nevertheless, it would be better than the current situation.

[rgooch]

Previous discussion on mailing list: https://groups.google.com/d/topic/golang-dev/wUEWhk2jtHM/

[minux]

This seems going backwards to manual memory management and it complicated the runtime a lot. I don't like this idea. As mentioned in another issue, the GC could do a stop the world emergency collection when memory runs out, and it should help the usual case where there still is free memory available but GC has yet to claim them. If the program indeed runs out of memory, then it isn't much that the panic handler could do to remedy the situation.

[rgooch]

Regarding the appearance of going backwards: it's not really the same as old-school memory management. Go would still be a fully garbage collected language, with all the conveniences that entails. Further, most people could just ignore the whole issue and keep playing the "I'm feeling lucky" game. Nothing changes for them.

Regarding making the runtime more complex: are you sure about that? What I'm suggesting is pretty narrow in scope. No "special memory" regions.

Regarding the emergency GC run: I haven't seen the proposal. If the suggestion is coupled with a recoverable panic(), so that the programme has the necessary feedback that memory is running out, this approach may help in many cases.

Regarding the panic handler not being able to do much to remedy an OOM situation: that is not correct. There are many classes of problems where there is a fairly high-level point at which an entire transaction can be aborted. For example, if I get a transaction request via an RPC handler then I would put the work which allocates lots of objects into a function and place a panic handler in there. The handler would drop the references to the new objects and call runtime.GC(). If the function returns a status indicating an OOM was caught, a failure status would be send back in the RPC reply, and I could even apply some back-pressure on requests.

It's essential to have an effective feedback mechanism when allocating memory. Without that, an application which dynamically scales its resource usage has no robust way to throttle.

[aclements]

(This may morph in to a runtime issue, but for now I think it's more of a proposal.)

[aclements]

I understand the desire for a memory backpressure mechanism, but I see some issues with the proposed solution.

For one, modern Linux systems (and probably other platforms) tend to make this difficult at the OS level itself. In practice, mmap operations generally let you overcommit memory (in fact, we depend on this on amd64) and things don't fail until you attempt to fault too much memory in. But at that point, the OS doesn't have much recourse but to kill your process. It is possible to configure Linux to disallow overcommit via /proc/sys/vm/overcommit_memory, though that tends to interfere with unexpected things (it may interfere with our overcommit, though that may be fixable). But maybe you have to disable overcommit for systems that depends on memory backpressure in any language.

Second, it's unclear what should happen if the failing allocation happens to be in a system goroutine. System goroutines don't allocate much, but they do allocate. Even the garbage collector has to allocate a bit for its work lists. Performing an immediate STW GC when allocation fails handles this situation much better, since it's a global solution to the global problem of an application being out of memory. A similar implementation-related problem is that if an allocation fails on a user goroutine but in the runtime, it's often on the "system stack" and making it possible to unwind a panic from the system stack would be a huge effort.

Finally, even if your defer releases reference to the memory used by a transaction, that memory won't be freed until the next GC, so allocations will continue to fail. Do you perform a full GC both before and after handling the panic? Do you require the recovery handler to perform a full GC?

[rgooch]

1: I often run systems with overcommit off, because I would rather have the (C) application "run out" of memory a little earlier than have it killed when it starts using memory it was promised.

2: If emergency stop-the-world garbage collection were to provide feedback to the application, this might be viable. If the feedback is in the form of a function the application can call to detect that STW GC was performed, it would have to be evaluated to determine how effective it is in practice. If there is not enough memory that can be garbage collected to fulfill the memory allocations before the next check, there will still be a panic. Some obvious examples would be a large make/append of a slice or a call to one of the encoders. On the other hand, if the feedback mechanism is to generate a recoverable panic(), it gives the application the ability to stop offending allocations even inside library code.

3: If you read my opening email, I stated that the application recovery code would probably need to call runtime.GC().

[minux]

How could you estimate the size for reserved memory? Please note that OOM can happen: 1. on any goroutine 2. on multiple goroutines concurrently 1 implies that unless you control every function call in your program, it's unlikely having reserved memory will help in every possible OOM case. Even worse, if the OOM handler is not last deferred function at all times, it's possible that OOM panic occurs and a later deferred cleanup function allocate more memory than is available, and then the program crashes. so deferring a OOM handler won't work in all cases. Or perhaps the program can register a global OOM handler, but that won't work either, because there is no way the global handler can release the problematic memory allocation that triggered the OOM panic. Also note that you can't control every possible case, for example, the runtime goroutine might allocate memory, and what if that triggers the OOM? 2 implies that it's very hard to generally estimate the memory required for OOM recovery. Unless you always only run only one goroutine, but that's not what typical Go programs are written. And I don't think we should add a mechanism that can only be reliably used for single goroutine programs.

[rgooch]

I think you're making this more complicated than it needs to be. It's up to the application programmer to pick the size of the reserved memory. Well-written recovery code is going to take very little memory. All it should do is drop references to data structures, call runtime.GC() and then re-reserve memory.

Control over every single function call isn't necessary. It's a common pattern that there are a small number of goroutines doing "housekeeping", and one or more goroutines doing the heavy transactions, and those are the ones that need to throttle. The reserved memory allows the housekeeping code to continue. All we need is a way for goroutines (or functions) to register that they want to handle an OOM panic. The heavy allocation functions register this interest and no-one else does. When the initial OOM occurs, the functions that registered interest will get the notification and stop what they're doing (just like with panic()), and all the others will continue, eating a little from the reserved memory.

I think that we can do this with a "soft panic", which only affects those functions who have registered. If registered, the notification has the same behaviour as a panic(). One possible API is to have a defer-like statement (perhaps onnotification) that a function can use to register interest. Every function up the call stack that registered would get the notification. Another possible API is a failifnotified() function which would return a bool indicating whether a notification was received while running the provided function.

[aclements]

1: I often run systems with overcommit off, because I would rather have the (C) application "run out" of memory a little earlier than have it killed when it starts using memory it was promised.

Interesting. Does Go have any problems running with overcommit disabled? It may be that large PROT_NONE mappings don't count against committed memory (which would make sense, but they do count against ulimit -v, which does not make sense).

2: If emergency stop-the-world garbage collection were to provide feedback to the application, this might be viable. ...

I don't think a recoverable panic is a technically viable solution for the reasons I mentioned in my comment (namely, allocations on system goroutines and system stacks), unless you're only interested in catching large failed allocations. So, what about the emergency GC approach? What sort of specific feedback would be useful to an application so it could scale to fit the resources in response to triggering emergency GCs? Would the latency of emergency GCs be a problem, or is taking a latency hit better than having the application die? Alternatively, could the runtime provide feedback or mechanisms that would be useful for staying under the memory limit, rather than trying to deal with the consequences once you've already run out (and is this only useful in a strictly single tenant system)?

3: If you read my opening email, I stated that the application recovery code would probably need to call runtime.GC().

I did read your opening email (how else would I have replied?). I have now re-read it a few times and I think your penultimate paragraph is proposing "yes" to both of my questions: that GC would happen twice during the recovery (both at the beginning and at the end), and that the second time would be triggered by the application recovery code after it had unwound and dropped references. Am I interpreting your proposal correctly?

[aclements]

Looks like our replies crossed.

All we need is a way for goroutines (or functions) to register that they want to handle an OOM panic. The heavy allocation functions register this interest and no-one else does.

I see. I think this is how you're proposing to get around failed allocations on system goroutines as well (though it's still a problem for failed allocations on system stacks). Still, if the failed allocation happens on a goroutine that isn't registered for OOM panics, what happens? Do you pick a random registered goroutine to take the panic? Do you trigger the panic no matter where the goroutine(s) is/are, even if it's not in allocation?

I'll have to think about this a bit. I'm still interested in exploring alternative mechanisms.

[minux]

I really don't think just handle OOM from allocation heavy goroutine/function would solve the problem. it's quite possible that the allocation heavy goroutine allocates the last bit of memory and a unrelated housekeeping function/goroutine then triggers the OOM. What does other GC backed language do to handle OOM gracefully? Does Java have OOM handlers?

[randall77]

Perhaps we're thinking about this the wrong way. What if instead of thinking about "what do we do at OOM", how about "how do we let the application figure out that it is getting close to OOM"?

I could see runtime functions something along the lines of currentMemoryInUse and maxMemoryAvailable. Heavy-allocating goroutines could check these two functions and backoff/fail the work instead of allocating.

Another possible API would be a "reserveMem(bytes int64) bool" call that tells the runtime "expect me to allocate this much". If the runtime thinks it will be too much, it can return false, meaning "probably not going to be able to allocate that much, please back off".

I'm less sure how one might implement those calls, especially the "when is the OS going to refuse my next allocation" question. It seems difficult in general, but we may be able to come up with something. Maybe when reserveMem(...) is called, we ensure that we can successfully map & commit the heap space required for that much memory?