You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Compression waits for the next fixed tick, so a just-closed chunk can sit uncompressed most of a day (81 GB observed); deadlock correlation to watch #1778
Two related field observations from the same production instance, same night, measured.
The scheduling gap
The twice-daily fixed compression schedule does not compress a chunk when it CLOSES - it waits for the next tick, which can be most of a day later. On the field box a query_stats chunk (2026-07-26, written entirely pre-#1767 while payload was still inline) grew to 81 GB before its scheduled compression ever reached it. Post-dedup chunks are far smaller so this severity was one-time, but the MECHANISM remains: the newest closed chunk is always the least-compressed data on disk for up to a tick interval, on every store.
Fix idea from the field: compress at chunk close (or a much shorter tick with a compress-if-closed check). Interacts with the compression-delay lever that shipped with #1768 - the delay governs eligibility age, the tick governs latency from eligibility to execution; this issue is about the tick.
The deadlock pattern (watch status - evidence, not yet a defect with a repro)
Four 40P01 deadlock_detected errors in one night's collection log, ALL self-healed on the very next cycle, no intervention:
22:37:29 procedure_stats - BEFORE any manual compression started
23:06:47 query_stats - ~5 min into a manual query_stats compression
00:35:04 query_stats - ~1h33m into the same compression
02:06:29 procedure_stats - ~10 min into a manual procedure_stats compression, i.e. the exact table being compressed at the time
3 of 4 landed inside active compression windows and the 4th predates them, so compression is correlated but not the sole trigger. Worth accreting future occurrences HERE so the pattern has a number. Design note carried from the field, worth taking seriously when changing the tick: if the deadlocks are genuinely compression-correlated, SHORTER and MORE FREQUENT compression windows (smaller chunks, compress-at-close) may reduce TOTAL exposure even at equal per-minute risk - the windows shrink faster than their count grows, post-dedup.
Also relevant to #1759 Phase 2 planning: measured compression throughput tops out ~16 MB/s on this host class (companion maintenance_work_mem issue) - use that number when budgeting any backfill/materialization pass on similar hardware.
The scheduling-gap half is fixed by #1779 (merged to dev at 47323ef): compression now runs on an hourly tick, and stores stuck on TimescaleDB's 12-hour default get converged - a just-closed chunk no longer sits uncompressed for most of a day. The deadlock half of the same night's observations got its own record: #1801, fixed by #1803 (deterministic digest ordering in the dimension batch upserts). If compression-correlated deadlocks show up again after both fixes, that is a fresh issue.
Activity
Merge dev into feature/1778-compression-tick
Merge dev into feature/1778-compression-tick (CHANGELOG only)
Merge pull request #1779 from erikdarlingdata/feature/1778-compressio…
erikdarlingdata commentedon Jul 28, 2026
The scheduling-gap half is fixed by #1779 (merged to dev at 47323ef): compression now runs on an hourly tick, and stores stuck on TimescaleDB's 12-hour default get converged - a just-closed chunk no longer sits uncompressed for most of a day. The deadlock half of the same night's observations got its own record: #1801, fixed by #1803 (deterministic digest ordering in the dimension batch upserts). If compression-correlated deadlocks show up again after both fixes, that is a fresh issue.
Compress on an hourly tick, and converge stores stuck on TimescaleDB'…
Phase 2 (erikdarlingdata#1759): --backfill-rollups, a preflighted ope…
Gate the compression phase on the hypertable's schema, not its bare name
Fall back to the cadence-only read when the job catalog has no phase …
Prove the degradation direction rather than describing it
Pin the compression policies to a phase grid clear of the refresh slo…
The twenty continuous aggregates join the compression ladder: compres…
Derive each raw hypertable's chunk interval from ingest (#4211) (#4332)
Reconcile each raw hypertable's chunk interval once a day from its in…