[API Proposal]: NoGC callback

[cshung]

Background and motivation

When people use the TryStartNoGCRegion method, in general, they are trying to prevent a GC from happening. However, this can be done as much as the allocation stays within the totalSize specified upfront.

Ensuring the application does not allocate more than totalSize is difficult, if not impossible. Right now, the GC will perform a GC when the totalSize is exceeded and the GC will have to garbage collect. This is okay from some applications, but for some others, it might not.

API Proposal

Customers are asking for a callback when that happens and let the customer decide if we should

1. Simply terminate the process as OutOfMemory.
2. Allocate more memory and let the process continue, or
3. Perform a GC and let the process continue.

We had some discussions (see below) to explore the actual use case and implementation constraints, and we discovered a key conflict between them.

From the requirements perspective, we would like to have a callback when the memory is exhausted so that we can gracefully terminate the game without a GC, but

From the implementation perspective, we cannot serve a callback that could potentially allocate more memory without a GC while we have already exhausted our pre-allocated memory.

To resolve this conflict, we notice that exactly exhausting the memory is not necessary from the requirement side. As long as we have a callback issued so that the game can gracefully terminate, this is good enough.

We also notice that nobody need the process to be terminated right away.

Therefore, I revised the proposal as follow:

API Usage

void OnBeforeEndNoGCRegion()
{
  /*
   * This callback will be invoked on the finalizer thread.
   * At this point, you have used 256M memory since you started the 
   * NoGCRegion, you have 4M to go before the NoGCRegion really ends
   * 
   * Since this is invoked on the finalizer thread, we expect minimal work here
   * to notify the game to terminate. The game loop thread will probably read
   * a flag and start showing goodbye.
   */
}

GC.TryStartNoGCRegion(260 * 1024 * 1024);
GC.RegisterAllocationCallback(256 * 1024 * 1024, OnBeforeEndNoGCRegion);
/*
 * Ideally, the work here should not use more than 256M, if that's the case, the code will 
 * perform a GC when EndNoGCRegion is invoked and that's the best case.
 * 
 * However, if the usage exceeds 256M, the callback will be invoked.
 * 
 * If the usage exceeds 260M, a GC will be automatically invoked and the NoGCRegion
 * will be terminated automatically.
 */
GC.EndNoGCRegion(); // And the callback will be detached automatically.

Alternative Designs

One alternative design is to expose a enum what to do when the NoGCRegion is exhausted. The options are to terminate the NoGCRegion, to commit more memory, or to fail fast. This is decided to be not good enough because what customer really wanted to customizable logic to show some screen and end the game gracefully, none of those options allow them to do that safely.

Another alternative design is to add parameters to the TryStartNoGCRegion. That will work, but there are two reasons why we wanted to have separate APIs for starting to NoGCRegion and registering the callback.

1. There are already a few overloads on GC.TryStartNoGCRegion, and adding more parameters to it makes it looks really complicated, and
2. We are envisioning extending this API to support scenarios other than just NoGCRegions, as of the time of writing, there is no plan to implement that yet, but having this API shape allows us to do that when we wanted.

Risks

By asking the caller to provide two limits, the caller must estimate how much memory is necessary for serving the callback, which circles back to the initial question that it is not easy to estimate memory usage. The key idea here is that the callback is supposed to be some simple thing, as @WenceyWang suggested below, it will simply be switching a flag and using some preallocated resources. That hopefully ease the problem.

Implementation-wise, we will reserve up to the larger limit to begin the NoGCRegion. Therefore, there is no risk to issue the callback without GC. When we reach the larger limit, we will terminate the NoGCRegion regardless. That way we can maintain our implementation reliability.

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and motivation

When people use the TryStartNoGCRegion method, in general, they are trying to prevent a GC from happening. However, this can be done as much as the allocation stays within the totalSize specified upfront.

Ensuring the application does not allocate more than totalSize is difficult, if not impossible. Right now, the GC will perform a GC when the totalSize is exceeded and the GC will have to garbage collect. This is okay from some applications, but for some others, it might not.

API Proposal

Customers are asking for a callback when that happens and let the customer decide if we should

1. Simply terminate the process as OutOfMemory.
2. Allocate more memory and let the process continue, or
3. Perform a GC and let the process continue.

To do that, we can add a mode parameter to the GC.TryStartNoGCRegion API to let the customer specify what behavior the user wanted. This optional mode parameter will be specified on all overloads of the method.

The mode parameter should be an enum of type NoGCRegionMode and have these values:

/**
 * This mode specify what the GC should do when we ran out of totalBytes within the NoGCRegion
 */
enum NoGCRegionMode
{
    /**
     * This means when we ran out of totalBytes allocated earlier, we will perform a GC.
     * This is the default behavior.
     */
    GC = 0, 
    /**
     * This means when we ran out of totalBytes allocated earlier, we will try to allocate more memory
     * from the operating system and let the application run without further GC.
     * 
     * This is not always feasible - in segments, if we ran out of the segment size, then we will fail.
     * or in regions, if we ran out of physical memory, we will fail as well.
     */
    ContinueAllocating = 1, 
    /**
     * This means we will let the application fail fast by throwing an OutOfMemoryException
     */
    Throws = 2,
}

### API Usage

```c#
GC.TryStartNoGCRegion(12345678, NoGCRegionMode.ContinueAllocating)

Alternative Designs

Since the user asked for a callback, it would be natural for us to provide a managed callback when that happens. We decided not to do that because it probably won't do what the user wants. Most likely, the user would like to trim some cache and give space for the application to continue to run at that point without incurring a GC. We have considered several implementation possibilities (e.g. serving a callback on allocation, throwing an OOM and catch, letting the user trim caches, etc ...), and nothing would achieve the goal that the user would like to. That is why we decided against the callback and offer the mode option instead.

Risks

No response

Author:	cshung
Assignees:	-
Labels:	api-suggestion, area-GC-coreclr, untriaged
Milestone:	-

[cshung]

@lucabol, this is meant to address #11733.
@WenceyWang, this is meant to address #37667.

[lucabol]

Yep, we have three requests now for basically the same thing. Let's hope it gets in.

[Maoni0]

this is the API proposal from the GC team :) please do let us know if the proposed API addresses your concerns. we'll give it some time for discussion then take it to the API review.

[lucabol]

Sorry @cshung @Maoni0 . I didn't realize this was the proposal :-)

This is a good proposal. Can you give more details on why exactly the callback wouldn't work? As it is the more general solution, it would be good to know why not.

BTW: could the user catch the OutOfMemoryException to achieve something similar to the callback?

[cshung]

Can you give more details on why exactly the callback wouldn't work?

Let's consider case 1, where we are attempting to save the current execution by allocating more memory. Remember we discovered the situation that we have already run out of preallocated memory, so we really cannot run more code before actually allocating more. Even if we do allocate more just to serve the callback, and assuming the amount of memory fits the callback's need, where would the callback be called? On the current thread? It would have a weird stack that an allocation suddenly calls another method. On a thread pool thread? What to do with the current thread right now? That (and any other threads that might allocate) cannot proceed. What if we ran out of memory again during the execution of the callback. This just turns into a lot of technical issues that we simply don't have answers to.

Now let's consider case 2, where we give up the current execution and throws an OOM on the spot. Now one could catch the OOM, but what can that exception handler do? First of all, it cannot allocate more memory (as it will fail right away again), so that excluded a lot of useful things that a user would like to do. One might be able to set null to certain references (e.g. global caches) in an attempt to have more memory to use, but even in that case, that memory won't be usable until another GC kicks in. Last but not least, an OOM can be thrown virtually anywhere. After you caught the OOM, are you sure you can recover from all possible partial states the process left off before the OOM happen? That would be a challenge for many people.

It is much easier to hard code these options and let the native runtime do it. Much of the limitation above comes from the fact that the processing of the situation cannot involve allocating more managed memory, this can easily be side-stepped by doing it in the native runtime. Of course, we lose generality, but do we really need it? Let's flip the question around. If you had the callback, what would you do with it - given you know all these limitations?

[lucabol]

In the most common case, I think, we can consider running out of totalSize as a programmer error (aka bug). Hence, we don't need to bring back the program in a running state, but just perform whatever application-specific 'bug reporting' is needed and shut down.

I think that is solved by catching the OOM exception. That was the original intent of the callback. So I think I am good with it.

[WenceyWang]

I think this is good, but I think we should clarify what does ContinueAllocating does. What does the totalBytes means if I use ContinueAllocating? Is GC will try to allocate a block of totalBytes again?

And what if the allocation failed? GC or OOM?

Or we can let it be 2 enum values ContinueAllocatingUntilThrow, which throw oom while allocating failed, and ContinueAllocatingThenGC for a GC while allocating failed?

[jkotas]

Now one could catch the OOM, but what can that exception handler do?

JITing methods that is going to happen as side-effect of running the exception handlers would likely cause some managed allocations too. For example, the JIT allocates string literals on the GC heap. These allocations would keep failing too.

I think that the NoGCRegionMode.Throws option is effectively equivalent to terminating the process ungracefully. It is unlikely that the application would be able to run much managed code to react to the OutOfMemoryException.

If we want to allow the application to react to running out of no GC region, I think a callback that is delivered on finalizer or threadpool thread in combination with allowing more allocation to succeed is the best option.

[cshung]

Or we can let it be 2 enum values ContinueAllocatingUntilThrow, which throw oom while allocating failed, and ContinueAllocatingThenGC for a GC while allocating failed?

I was thinking about ContinueAllocating behaving as if ContinueAllocatingUntilThrow, that is, when the application wants more memory than totalBytes, it will ask the operating system for more memory (*) to serve the request without raising a GC. At a certain point, it will reach a limit where we cannot do that anymore (**), and then it will raise an OOM, and the application crashes just like a typical OOM.

We don't have a ContinueAllocatingThenGC option, but we have a GC option, which behaving like GCThenContinueAllocating. When the application wants more memory than totalBytes, it will do a GC on the spot, effectively stopped the NoGCRegion, and then let the application goes on. This is the default option and is also the behavior we have today.

@WenceyWang, what does ContinueAllocatingThenGC means for you? Is that a mode you needed but not available here?

(*) Exactly how much we ask the operating system to serve the request is an unimportant implementation detail. All we needed is to make sure the allocation succeed, and therefore it is not necessary for us to expose that implementation detail.

(**) Without actually getting our hands on the implementation of this proposal, I am not sure exactly what does it mean by that limit. In the case of segments (which is the current mode right now), we cannot extend Gen0 larger than a segment, so the ContinueAllocating mode would probably stop when we reach the segment size. We might be able to do something to get around that, but we haven't investigated that possibility yet. In the case of the region, we should be able to allocate as much as we have physical memory.