Unexpected OOM with GCHeapLimit

[kevingosse]

Description

I run a code that allocates a 100MB array, iterates over it, then discard it (initially meant as a benchmark). When running with DOTNET_GCHeapHardLimit=7D000000, the code crashes at the 20th iteration with an OutOfMemoryException.

There are a number of things that make me think it's a bug:

• It only happens with regions
• It doesn't happen if I force a GC after each iteration
• It doesn't happen if I increase the size of the array to be 150MB or bigger
• When running with a debug CLR, it randomly triggers an assertion:

Assert failure(PID 30596 [0x00007784], Thread: 12248 [0x2fd8]): planned_regions_per_gen[0]
    File: E:\git\runtime\src\coreclr\gc\gc.cpp Line: 31714
    Image: E:\git\runtime\artifacts\bin\coreclr\windows.x64.Debug\corerun.exe


Assert failure(PID 30596 [0x00007784], Thread: 18508 [0x484c]): saved_planned_regions_per_gen[0] >= to_be_empty_regions
    File: E:\git\runtime\src\coreclr\gc\gc.cpp Line: 31721
    Image: E:\git\runtime\artifacts\bin\coreclr\windows.x64.Debug\corerun.exe


Assert failure(PID 30596 [0x00007784], Thread: 16148 [0x3f14]): saved_planned_regions_per_gen[0] >= to_be_empty_regions
    File: E:\git\runtime\src\coreclr\gc\gc.cpp Line: 31721
    Image: E:\git\runtime\artifacts\bin\coreclr\windows.x64.Debug\corerun.exe


Assert failure(PID 30596 [0x00007784], Thread: 16720 [0x4150]): saved_planned_regions_per_gen[0] >= to_be_empty_regions
    File: E:\git\runtime\src\coreclr\gc\gc.cpp Line: 31721
    Image: E:\git\runtime\artifacts\bin\coreclr\windows.x64.Debug\corerun.exe


Assert failure(PID 30596 [0x00007784], Thread: 17976 [0x4638]): saved_planned_regions_per_gen[0] >= to_be_empty_regions
    File: E:\git\runtime\src\coreclr\gc\gc.cpp Line: 31721
    Image: E:\git\runtime\artifacts\bin\coreclr\windows.x64.Debug\corerun.exe


Assert failure(PID 30596 [0x00007784], Thread: 37468 [0x925c]): saved_planned_regions_per_gen[0] >= to_be_empty_regions
    File: E:\git\runtime\src\coreclr\gc\gc.cpp Line: 31721
    Image: E:\git\runtime\artifacts\bin\coreclr\windows.x64.Debug\corerun.exe


Assert failure(PID 30596 [0x00007784], Thread: 35856 [0x8c10]): saved_planned_regions_per_gen[0] >= to_be_empty_regions
    File: E:\git\runtime\src\coreclr\gc\gc.cpp Line: 31721
    Image: E:\git\runtime\artifacts\bin\coreclr\windows.x64.Debug\corerun.exe


Assert failure(PID 30596 [0x00007784], Thread: 28252 [0x6e5c]): saved_planned_regions_per_gen[0] >= to_be_empty_regions
    File: E:\git\runtime\src\coreclr\gc\gc.cpp Line: 31721
    Image: E:\git\runtime\artifacts\bin\coreclr\windows.x64.Debug\corerun.exe


Assert failure(PID 30596 [0x00007784], Thread: 37336 [0x91d8]): saved_planned_regions_per_gen[0] >= to_be_empty_regions
    File: E:\git\runtime\src\coreclr\gc\gc.cpp Line: 31721
    Image: E:\git\runtime\artifacts\bin\coreclr\windows.x64.Debug\corerun.exe


Assert failure(PID 30596 [0x00007784], Thread: 33332 [0x8234]): saved_planned_regions_per_gen[0] >= to_be_empty_regions
    File: E:\git\runtime\src\coreclr\gc\gc.cpp Line: 31721
    Image: E:\git\runtime\artifacts\bin\coreclr\windows.x64.Debug\corerun.exe


Assert failure(PID 30596 [0x00007784], Thread: 44380 [0xad5c]): saved_planned_regions_per_gen[0] >= to_be_empty_regions
    File: E:\git\runtime\src\coreclr\gc\gc.cpp Line: 31721
    Image: E:\git\runtime\artifacts\bin\coreclr\windows.x64.Debug\corerun.exe

Note: the assertion doesn't happen every time, so it could be unrelated.

Reproduction Steps

Run the following code:

  public static long Allocate()
  {
      var array = new byte[100 * 1024 * 1024];

      long sum = 0;

      for (int i = 0; i < array.Length; i++)
      {
          sum += array[i];
      }

      return sum;
  }

static void Main()
{
    for (int i = 0; i < 10_000; i++)
    {
        Allocate();
    }
}

With:

DOTNET_GCHeapHardLimit=7D000000
DOTNET_gcServer=1

My machine has 12 cores and it seems to affect the problem, so you might have to adjust the number of heaps accordingly.

Expected behavior

The code runs without crashing.

Actual behavior

An OOM is thrown after 19 iterations.

Regression?

The problem seems related to regions. It happens in .NET 7 and 8, but I can't reproduce it in .NET 6 or when using clrgc.dll.

Known Workarounds

Use clrgc.dll or increase the heap limit.

Configuration

.NET 8 RC2
Windows x64

Other information

No response

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

I run a code that allocates a 100MB array, iterates over it, then discard it (initially meant as a benchmark). When running with DOTNET_GCHeapHardLimit=7D000000, the code crashes at the 20th iteration with an OutOfMemoryException.

There are a number of things that make me think it's a bug:

• It only happens with regions
• It doesn't happen if I force a GC after each iteration
• It doesn't happen if I increase the size of the array to be 150MB or bigger
• When running with a debug CLR, it triggers an assertion:

Assert failure(PID 30596 [0x00007784], Thread: 12248 [0x2fd8]): planned_regions_per_gen[0]
    File: E:\git\runtime\src\coreclr\gc\gc.cpp Line: 31714
    Image: E:\git\runtime\artifacts\bin\coreclr\windows.x64.Debug\corerun.exe


Assert failure(PID 30596 [0x00007784], Thread: 18508 [0x484c]): saved_planned_regions_per_gen[0] >= to_be_empty_regions
    File: E:\git\runtime\src\coreclr\gc\gc.cpp Line: 31721
    Image: E:\git\runtime\artifacts\bin\coreclr\windows.x64.Debug\corerun.exe


Assert failure(PID 30596 [0x00007784], Thread: 16148 [0x3f14]): saved_planned_regions_per_gen[0] >= to_be_empty_regions
    File: E:\git\runtime\src\coreclr\gc\gc.cpp Line: 31721
    Image: E:\git\runtime\artifacts\bin\coreclr\windows.x64.Debug\corerun.exe


Assert failure(PID 30596 [0x00007784], Thread: 16720 [0x4150]): saved_planned_regions_per_gen[0] >= to_be_empty_regions
    File: E:\git\runtime\src\coreclr\gc\gc.cpp Line: 31721
    Image: E:\git\runtime\artifacts\bin\coreclr\windows.x64.Debug\corerun.exe


Assert failure(PID 30596 [0x00007784], Thread: 17976 [0x4638]): saved_planned_regions_per_gen[0] >= to_be_empty_regions
    File: E:\git\runtime\src\coreclr\gc\gc.cpp Line: 31721
    Image: E:\git\runtime\artifacts\bin\coreclr\windows.x64.Debug\corerun.exe


Assert failure(PID 30596 [0x00007784], Thread: 37468 [0x925c]): saved_planned_regions_per_gen[0] >= to_be_empty_regions
    File: E:\git\runtime\src\coreclr\gc\gc.cpp Line: 31721
    Image: E:\git\runtime\artifacts\bin\coreclr\windows.x64.Debug\corerun.exe


Assert failure(PID 30596 [0x00007784], Thread: 35856 [0x8c10]): saved_planned_regions_per_gen[0] >= to_be_empty_regions
    File: E:\git\runtime\src\coreclr\gc\gc.cpp Line: 31721
    Image: E:\git\runtime\artifacts\bin\coreclr\windows.x64.Debug\corerun.exe


Assert failure(PID 30596 [0x00007784], Thread: 28252 [0x6e5c]): saved_planned_regions_per_gen[0] >= to_be_empty_regions
    File: E:\git\runtime\src\coreclr\gc\gc.cpp Line: 31721
    Image: E:\git\runtime\artifacts\bin\coreclr\windows.x64.Debug\corerun.exe


Assert failure(PID 30596 [0x00007784], Thread: 37336 [0x91d8]): saved_planned_regions_per_gen[0] >= to_be_empty_regions
    File: E:\git\runtime\src\coreclr\gc\gc.cpp Line: 31721
    Image: E:\git\runtime\artifacts\bin\coreclr\windows.x64.Debug\corerun.exe


Assert failure(PID 30596 [0x00007784], Thread: 33332 [0x8234]): saved_planned_regions_per_gen[0] >= to_be_empty_regions
    File: E:\git\runtime\src\coreclr\gc\gc.cpp Line: 31721
    Image: E:\git\runtime\artifacts\bin\coreclr\windows.x64.Debug\corerun.exe


Assert failure(PID 30596 [0x00007784], Thread: 44380 [0xad5c]): saved_planned_regions_per_gen[0] >= to_be_empty_regions
    File: E:\git\runtime\src\coreclr\gc\gc.cpp Line: 31721
    Image: E:\git\runtime\artifacts\bin\coreclr\windows.x64.Debug\corerun.exe

Reproduction Steps

Run the following code:

  public static long Allocate()
  {
      var array = new byte[100 * 1024 * 1024];

      long sum = 0;

      for (int i = 0; i < array.Length; i++)
      {
          sum += array[i];
      }

      return sum;
  }

static void Main()
{
    for (int i = 0; i < 10_000; i++)
    {
        Allocate();
    }
}

With:

DOTNET_GCHeapHardLimit=7D000000
DOTNET_gcServer=1

My machine has 12 cores and it seems to affect the problem, so you might have to adjust the number of heaps accordingly.

Expected behavior

The code runs without crashing.

Actual behavior

An OOM is thrown after 19 iterations.

Regression?

The problem seems related to regions. It happens in .NET 7 and 8, but I can't reproduce it in .NET 6 or when using clrgc.dll.

Known Workarounds

Use clrgc.dll or increase the heap limit.

Configuration

.NET 8 RC2
Windows x64

Other information

No response

Author:	kevingosse
Assignees:	-
Labels:	area-GC-coreclr
Milestone:	-

[mangod9]

@cshung @mrsharm can you please try to repro?

[Maoni0]

thanks for reporting this!

the asserts you were saw are not related to the premature OOM. although I'm also interested in these asserts but I can't seem to repro locally so if you wouldn't mind getting a dprintf log that enables level 1 and REGIONS_LOG that'd be really helpful.

the premature OOM is because regions is too aggressive at decommitting free huge regions (ie, regions larger than the standard size for LOH) - in blocking GCs we always move these regions to the decommit list as long as the allocation budget doesn't require us to keep them. and in this case because all the LOH objects are temporary, the alloc budget for LOH is tiny so we always move them to the decommit list. and decommit does not happen so fast so we throw OOM when most of the memory is still on the decommit list.

I have a temp fix at https://github.com/Maoni0/runtime/tree/94175_fix if you want to give it a try - the idea is we also apply age based policy to these free huge regions like we do with the non huge regions (but with a different policy).

[kevingosse]

Sorry for the long wait. I confirm that I get no more OOM with your fix.

There was a compilation error though:

 E:\git\runtime\src\coreclr\gc\gc.cpp(4335): error C2248: 'WKS::region_free_list::get_region_kind': cannot access priv
  ate member declared in class 'WKS::region_free_list'

(I just made the field public to fix it)

For the assertion, unfortunately I'm not getting it anymore. I don't know what has changed. Anyway, if I see them again I'll collect the logs and send them.

[mangod9]

@Maoni0, were you able to get your presumptive fix merged?

[markples]

Thank you @kevingosse for the bug report. This should be fixed now.