Skip to content

Compression waits for the next fixed tick, so a just-closed chunk can sit uncompressed most of a day (81 GB observed); deadlock correlation to watch #1778

Closed

Description

@erikdarlingdata

Two related field observations from the same production instance, same night, measured.

The scheduling gap

The twice-daily fixed compression schedule does not compress a chunk when it CLOSES - it waits for the next tick, which can be most of a day later. On the field box a query_stats chunk (2026-07-26, written entirely pre-#1767 while payload was still inline) grew to 81 GB before its scheduled compression ever reached it. Post-dedup chunks are far smaller so this severity was one-time, but the MECHANISM remains: the newest closed chunk is always the least-compressed data on disk for up to a tick interval, on every store.

Fix idea from the field: compress at chunk close (or a much shorter tick with a compress-if-closed check). Interacts with the compression-delay lever that shipped with #1768 - the delay governs eligibility age, the tick governs latency from eligibility to execution; this issue is about the tick.

The deadlock pattern (watch status - evidence, not yet a defect with a repro)

Four 40P01 deadlock_detected errors in one night's collection log, ALL self-healed on the very next cycle, no intervention:

  • 22:37:29 procedure_stats - BEFORE any manual compression started
  • 23:06:47 query_stats - ~5 min into a manual query_stats compression
  • 00:35:04 query_stats - ~1h33m into the same compression
  • 02:06:29 procedure_stats - ~10 min into a manual procedure_stats compression, i.e. the exact table being compressed at the time

3 of 4 landed inside active compression windows and the 4th predates them, so compression is correlated but not the sole trigger. Worth accreting future occurrences HERE so the pattern has a number. Design note carried from the field, worth taking seriously when changing the tick: if the deadlocks are genuinely compression-correlated, SHORTER and MORE FREQUENT compression windows (smaller chunks, compress-at-close) may reduce TOTAL exposure even at equal per-minute risk - the windows shrink faster than their count grows, post-dedup.

Also relevant to #1759 Phase 2 planning: measured compression throughput tops out ~16 MB/s on this host class (companion maintenance_work_mem issue) - use that number when budgeting any backfill/materialization pass on similar hardware.

Activity

  1. erikdarlingdata commented on Jul 28, 2026

    @erikdarlingdata
    OwnerAuthor

    The scheduling-gap half is fixed by #1779 (merged to dev at 47323ef): compression now runs on an hourly tick, and stores stuck on TimescaleDB's 12-hour default get converged - a just-closed chunk no longer sits uncompressed for most of a day. The deadlock half of the same night's observations got its own record: #1801, fixed by #1803 (deterministic digest ordering in the dimension batch upserts). If compression-correlated deadlocks show up again after both fixes, that is a fresh issue.

  2. added 2 commits that reference this issue on Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions