Schema registry uses ~4x more memory on 1.7 than on 1.6, OOM-killing environments with many deployments

[rielas]

After migrating to 1.7 we faced a few OOMs related to the size of the Restate registry (deployments + versions). A registry that started without problems on 1.6 surprisingly started consuming several times more memory on 1.7. This led to several incidents on free-tier accounts.

We managed to reproduce the issue e2e — code, example run.

Basically, it registers a critical number of deployments (50 deployments x 50 services, ~52 MiB registry), starts successfully on the old version with a 1 GiB memory limit, and then after the upgrade to 1.7 it crashes. In the linked run, byte-identical data starts on 1.6.2 at 1 GiB (peak 665 MiB), is OOM-killed on 1.7.3 at the same limit, and fits again at 2 GiB — live heap on identical data goes 246 MiB -> 997 MiB, a ~4x regression.

Related incidents

• INC-445 — env-201kbhtqassagmd9t46x1s2sebq
• INC-453 — env-201kz5yreswh13k01nr41xrfprp
• INC-454 — env-201keeqwd1b6dp2sywwn5j6qq7k and env-201kbw5cf3edzz2v3dxerm4x61v
• INC-461 — env-201kb5432yzqxafv3xawvzd1md3

cc @AhmedSoliman @muhamadazmy @pcholakov

[tillrohrmann]

Thanks for reporting this issue @rielas. Since v1.6.2 there were really only very few changes made to the metadata server that could explain the behavioral difference. The main ones being that we updated the raft-rs dependency and changed a little bit how RocksDB is configured but I have a hard time explaining this difference with these changes.

(50 deployments x 50 services, ~52 MiB registry)

I thought we were limiting the maximum size of the metadata to be 32 MiB. Having such a large schema metadata is certainly not good and can cause some problems (high memory usage) because we are keeping the schema information in memory.

In v1.7.0 we started storing the schema information in the partition processor which requires a self proposal of this change. Until it's written to Bifrost the schema copy is kept in memory. So I believe that this is causing the increased memory usage.

How many partitions are you running in these environments?

[rielas]

How many partitions are you running in these environments?

8

[rielas]

Claude points to #5000 as the confirmed cause

[tillrohrmann]

I think there is not a lot we can short-term wise do other than increasing the memory. It's not a bug but an artifact of having a schema information that is far too large. Is it possible to make users aware of it and ask them to clean up old deployments?

We have plans to rework the schema information and to represent it more efficiently but this is not a short-term fix.

[pcholakov]

We could certainly look at surfacing unused drained deployments information to users. We've mentioned previously that automated pruning support for those may one day be in scope for restate itself - is that at all on the horizon? We've seen that the introspection SQL queries themselves can impose significant runtime costs so native support for pruning, or at least efficiently enumerating stale deployments, would be very much desirable here.

[muhamadazmy]

I know this is orthogonal to this issue but I am working on a change to use bilrost encoding for Schema object. This should reduce the size of the schema object. Another change that I am planning to do is to also deduplicate "json schema" definitions for services inputs and outputs, which is using the most sizes specially across deployments.

[rielas]

As mitigations, we could try:

1. Before migrating to 1.7, estimate and update memory limits based on registry size (the dependency is pretty linear)
2. Configure VPA to raise memory limits on OOMs
3. Add a button in Restate Cloud that runs a workflow to delete old drained deployments and reports how much memory was freed

[MohamedBassem]

Ok, so I've been looking a bit into that. Let's start with some of the facts:

1. The synthetic schema here serializes to around 25MB on the wire with flexbuffers encoding (so it's within the 32MB max record size).
2. The free tier (before the recent changes) has a 1GB ram limit, and it's configured with the following:
   • 256MB rocksdb memory
   • 256MB invoker memory
   • 64MB query engine memory
   • 250MB record cache size (the default).
   • 8 partitions

I've asked claude to take memory profiles for 1.6.2 and 1.7.3 after the memory settles to compare where the inflation is coming from, here's the flamegraph that claude visualised: https://claude.ai/code/artifact/6693d774-4968-468d-9d79-a9c810baa11b

In 1.6.2, the memory settled on ~240MB, 136MB of which are in the memory representation of the schema! But this is reasonable because the node wasn't doing anything else.

The story in 1.7.3 is very different though. In 1.7.3 memory peaked at 1981MB, and settled at 1032MB. The heap dump is from the settled memory, and it's already above the 1GB limit that we have. In 1.7.X we started upserting the schema to all partitions in parallel which happens on startup, this lead to the following memory to be allocated:

1. 224MB (unaccounted for memory) coming from the serialization buffers that we use for partition store. We have 8 partitions, so that's around 28MB per partition.
2. 448MB (unaccounted for memory) coming from rocksdb's WriteBatchWithIndex. It seems like in 1.7.0 we started resuing the WBI (Reuse writebatch buffers in state machine #4782), which means that oversized records (like the one of the schema), inflates the internal buffers of the WBI. That's 56MB per partition (that's the effect of capacity doubling in c++ containers). Before 1.7.0, that would have been a spike, that then gets freed when the WBI is destroyed. So it'd still have been bad, but wouldn't be sustained memory growth.
3. 150MB in memory representation of the schema object, similar to 1.6.2.

And this is just the "settled" memory. During the upsert spike on startup, there are more buffers that get allocated (e.g. the background appender arena, the record cache bytes clone), the memtable memory, etc. So it's very easy to hit the 1GB (even a 2GB limit tbh).

Now, we can blame this on the schema upsert change. However, to be honest, this exact problem can happen if a customer is writing equally large records, and the service would end up OOMing exactly the same way. If we truely want to keep the 1GB limit, we should instead perhaps consider decreasing the max record size on the free tier to maybe something like 10MB.

So my recommendation would be:

1. Either accept the 2GB for free tier, or reduce max record size (though that can only be safely applied to new envs).
2. Abandon ([metadata] Add a configurable limit for the number of registered deployments #5189). Restricting the number of deployments means little here. The problem is in the metadata size, which can be arbitrary long per service. So whatever number we restrict here, is kinda irrelevant as you can get arbitrary large metadata from one deployment. Reducing max record size would put a natural limit on metadata size. Or change my PR to be on the size rather than the count (though it becomes less intuitive to the users).
3. The schema is heavily compressible (most of the size is coming from the JSON schema metadata). So while @muhamadazmy is working on moving away from flexbuffers to bilrost, perhaps compression can be a faster bandaid. We can maybe ship an experimental flag in 1.8 that enables compression (and forfeits the right to downgrade to 1.7). @AhmedSoliman @muhamadazmy what do you think?
4. As for the buffer inflations (WBI and partition store arenas), those only grow up and would never shrink. Which is fine if the env is always dealing with large records, but not great for one of records like this case. I don't know what's a good solution here, and what makes things worse is that they're unaccounted for completely. In some of our large envs with 30 partitions per node, that's ~2.5GB of unaccounted for memory. Maybe we can reset those buffers after large encodes (where "large" is determined by some sort of a coarse record histogram?), it doesn't help with spikes though.
5. The coordinated upsert schema on startup is problematic, but as I mentioned, it can also happen with user-controlled large records. We can try to jitter it as a best effort thing optimization, but I'm also ok with doing nothing about it as long as we account correctly for the memory spike it introduces.
6. Although this is irrelevant in that issue, the fact that the schema can be 26MB encoded, but 140MB in memory representation makes me believe that the record cache accounting that we do can be way off. We keep PolyBytes::Typed, but consume from the cache its encoded size. Luckily, records don't stay in the record cache for long.