Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: consi/flowcus
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v0.10.2
Choose a base ref
...
head repository: consi/flowcus
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: v0.10.3
Choose a head ref
  • 2 commits
  • 7 files changed
  • 1 contributor

Commits on Mar 26, 2026

  1. New screenshot

    @consi
    consi committed Mar 26, 2026
    Configuration menu
    Copy the full SHA
    04c92bc View commit details
    Browse the repository at this point in the history

  2. feat: streaming merge engine with bounded memory, work queue, and ada… 

    …ptive throttle

Redesign background merge to prevent OOM on large hour partitions:

- Batch-limited merge plans: max 8 parts per batch (configurable), multiple
  rounds converge via generation increments instead of one mega-merge
- Streaming column-at-a-time execution: k-way merge order from rowids, then
  process each column independently — peak memory drops from all-columns to
  one-column plus merge order array
- PartWriter: incremental .inprogress directory pattern for both ingestion
  and merge, atomic rename on completion, crash-safe cleanup on startup
- Persistent worker threads with bounded work queue replacing per-plan
  spawn_blocking with semaphore
- Exponential throttle: halve slots on high CPU/mem, double on recovery,
  minimum 1 — running merges never interrupted
- Default scan interval 30s (was 15s)
    @consi
    consi committed Mar 26, 2026
    Configuration menu
    Copy the full SHA
    95f2c39 View commit details
    Browse the repository at this point in the history
Loading