High Memory usage due to SlabMemoryPool

[alkampfergit]

Dear Orleans Team,

We are in .NET 9 Orleans 9.0.1

we have a software that is running with Microsoft orleans, actually it is a single node instance and in one of the installation we started seeing anomaly RAM usage, the process start using several GB of RAM.

Doing a dump we found that we have more than one millions instances of Orleans.Networking.Shared.MemoryPoolBlock

And following the Roots in the Heap we found that the class holding reference is SlabMemoryPool. We look in the heap and we found 35k instances of MemoryPoolSlab (each one using 128 k of memory)

Actually we are investigating on what could be the cause, but if I'm reading the code correctly the SlabMemoryPool will always allocate new slabs if there is not an available block in the pool, so the RAM usage grow and was never returned back. Supposing we surely have some problem somewhere else and we have a moment in which some piece of code ask for too many MemoryPoolBlock for the SlabMemoryPool the memory will never be given back to the garbage collector.

How can I better investigate the problem?

Is there a known workaround to free the memory? (Actually when the allocation is really high the server goes in out-of-memory-exception)

[AlgorithmsAreCool]

1. Is this still happening?
2. How long was the processing running before the memory began to grow out of control?
3. How many grain activations are you running?

[ReubenBond]

I'm fairly sure we fixed this in a subsequent release. I highly recommend using the latest (v9.2.1). I may be mistaken (currently not near a PC).

[ReubenBond]

The ability for memory pools to shrink was recently added to Kestrel (where the memory pool code originates): dotnet/aspnetcore#61554 - someone could port that PR over (replace the current mem pool with Kestrel's current mem pool, incl mem releasing)

[alkampfergit]

We are planning to upgrade the customer to latest version in the near future, actually the process (that usually was around 2GB of memory) after a couple of days of usage is usually 5 GB. This is not a problem until for some situation it grows out of control and reach almost 10GB.

The problem probably originate from the fact that we uses orleans on a legacy code, we are usin old NewtonsoftJson serialization because we have data on db that originates from 2015, so to avoid any problem we still use the old serialize. So we are aware that sometimes we have object that serialize to a big blob of json. We suspect that when some massive operation happens, we generate many of them so we could trigger the problem.

Also we serve with kestrel video, big file (part of our software handle documents, where people can load anything, we have file that are more than 1 GB).

I'm pretty sure that up to orleans 8 we do not have this problem ,this start happening when we move to .NET 9 and orleans 9.0.1

I'll keep you informed on what will happen when we will upgrade to the new version of orleans.

Thanks