compute/storage: Eliminate compute (re)hydration memory spikes

[ParkMyCar]

Internal tracking reference: https://github.com/MaterializeInc/database-issues/issues/7017

Overview

Rehydration of compute dataflows requires more memory than in the steady state. This is a frustrating customer experience, as they need to size their clusters and therefore pay for peak memory usage, even though the steady state memory usage may be much lower—sometimes even an order of magnitude lower.

In many cases, this behavior is "in spec". We don't today have a requirement like "peak memory usage must be within x% of steady state memory usage." But perhaps we should!

Possible work items

• Development of a specification for a limit on peak memory usage relative to steady state memory usage
  • Using disk-backed allocations in places that are rehydration sensitive
    • Using lgalloc in persist_source
    • Using lgalloc for consolidation in persist_sink: https://github.com/MaterializeInc/database-issues/issues/8464
  • Further tuning of persist_source backpressure: compute: use granular backpressure in persist_source decoding #22549

Activity log

• 12 June 2024. Subsumed by https://github.com/MaterializeInc/database-issues/issues/3748.

• 21 February 2024. Split out persist's Q1 work on this initiative into a dedicated tracking issue: https://github.com/MaterializeInc/database-issues/issues/7601. Kicked this issue back into compute's "Later" pile.