You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
feat(exemplar): disk budget arithmetic inside 8 GiB
Implements the frozen #201 contract.
Q1 - 8 GiB budget table documented in CLAUDE.md (4.5 GiB main relational
tier, 1.5 GiB aggregate.db, 0.5 GiB DLQ, 0.5 GiB WAL/temp, 1 GiB mandatory
headroom) with per-component measured high-water gauges.
Q2 - EXEMPLAR_BYTES_GLOBAL_WINDOW default 8 MiB -> 3 MiB. New
EXEMPLAR_RETENTION_DAYS=2 drives a separate transactional purge of exemplar
traces, spans, logs, their FTS rows and expired weak references, running
ahead of the HOT_RETENTION_DAYS purge on the same hourly tick. Aggregate
retention stays 7 days.
Q3 - synthesized logs are metered: every one reserves
len(body)+len(attributesJSON)+logRowFixedBytes against the selected trace's
per-trace budget and the shared per-service/global window budgets, under
EXEMPLAR_SYNTH_LOGS_PER_SPAN=8 and EXEMPLAR_SYNTH_LOGS_PER_TRACE=64.
Refusals drop the log, count synth_per_span|synth_per_trace|budget_bytes,
and stamp the trace truncated.
Q4 - reservation lifecycle replaces immediate charging: reserve before row
construction, commit when the primary queue or DLQ accepts the batch,
release only when the row never reached a destination. Reserved bytes bind
the cap. Bytes accepted downstream are never refunded on selection
eviction; count slots still are.
Q5 - disk watchdog with staged shedding and hysteresis. statfs on
DATA_DISK_PATH is the enforcement source, ceiling = min(DATA_DISK_BUDGET_MB,
usable volume capacity). >=90% admits only error exemplars; >=95% disables
all raw admission and the exemplar DLQ fallback, purges the expired
exemplar tier, checkpoints the WAL and fails readiness. Recovery needs
<90% and <85% respectively. Raw shedding never fails a successful Export;
an authoritative aggregate commit hitting ENOSPC/SQLITE_FULL does, with
RESOURCE_EXHAUSTED.
-`DEFAULT_TENANT` (`default`) — assigned to rows ingested without explicit tenant
371
371
-`HOT_RETENTION_DAYS` (7) — drives `RetentionScheduler`; range 1..36500
372
+
-`EXEMPLAR_RETENTION_DAYS` (2) — separate, shorter retention for the raw exemplar tier in aggregate mode; validated 1..`HOT_RETENTION_DAYS`. See the Data Disk Budget section
373
+
-`EXEMPLAR_BYTES_GLOBAL_WINDOW` (**3 MiB**, was 8 MiB) / `EXEMPLAR_BYTES_PER_SERVICE_WINDOW` (512 KiB) — instance-wide and per-service byte budget per 5-minute window
374
+
-`EXEMPLAR_SYNTH_LOGS_PER_SPAN` (8), `EXEMPLAR_SYNTH_LOGS_PER_TRACE` (64) — count caps on logs synthesized from span events and span status
375
+
-`DATA_DISK_BUDGET_MB` (8192), `DATA_DISK_PATH` (`./data`) — disk watchdog ceiling and the volume it `statfs`-es
-`METRIC_MAX_CARDINALITY` (10000), `METRIC_MAX_CARDINALITY_PER_TENANT` (0 = unlimited), `API_RATE_LIMIT_RPS` (100). The per-tenant cap is checked first; when set, a noisy tenant cannot exhaust the global pool. Overflow is labeled by tenant via `otelcontext_tsdb_cardinality_overflow_by_tenant_total{tenant_id}` (`__global__` sentinel when the global cap was the trigger).
Every log synthesized from a span event or span status reserves
507
+
`len(body) + len(attributesJSON) + logRowFixedBytes` against its trace's
508
+
per-trace budget AND the shared per-service/global window budgets, under
509
+
`EXEMPLAR_SYNTH_LOGS_PER_SPAN` and `EXEMPLAR_SYNTH_LOGS_PER_TRACE`. They do
510
+
**not** consume the ordinary log-exemplar quota — that budget is for logs a
511
+
client actually sent — but they are not weightless either: a span carrying two
512
+
hundred exception events used to write two hundred rows no budget had ever
513
+
seen. Refusals drop the log, count
514
+
`otelcontext_exemplar_dropped_total{signal="logs",reason}` with
515
+
`synth_per_span|synth_per_trace|budget_bytes`, and stamp the trace `truncated`.
516
+
517
+
#### Reservation lifecycle (no unconditional refunds)
518
+
519
+
Bytes are **reserved** before a row is constructed, **committed** when the
520
+
primary queue or the DLQ accepts the batch, and **released** only when the row
521
+
is dropped before submission or permanently lost because both destinations
522
+
refused it. Reserved bytes bind the cap exactly like committed ones. Once a
523
+
submission is accepted the charge is **monotonic for that window**: displacing
524
+
the trace later releases the count slot (a slot is a seat, not a byte) and
525
+
never the bytes — refunding bytes already on disk is how a window writes past
526
+
its cap. `Batch.Reservation` carries the charge to the submit boundary.
527
+
528
+
#### Staged shedding, hysteresis, and the one ENOSPC exception
529
+
530
+
| Volume usage | State | Behaviour |
531
+
|---|---|---|
532
+
| ≥ 90% |`errors_only`| Only error trace/log exemplars are admitted; healthy/slow/WARN raw retention off |
533
+
| ≥ 95% |`raw_off`| ALL new raw exemplar admission off, exemplar DLQ fallback closed, immediate expired-exemplar purge + `wal_checkpoint(TRUNCATE)`, `/ready` → 503 |
534
+
535
+
Hysteresis: recover from `raw_off` only below 90%, from `errors_only` only
536
+
below 85%. A failed `statfs` HOLDS the current state — shedding because a
537
+
syscall failed would be an outage caused by the safety mechanism.
538
+
539
+
Raw shedding **never** turns a successful aggregate Export into a retryable
540
+
failure. One exception: if the **authoritative** aggregate commit fails with
541
+
`ENOSPC` or `SQLITE_FULL` (`aggregate.IsDiskFull`), the Export MUST fail with
542
+
`RESOURCE_EXHAUSTED`/429 — under the durable-ACK contract a success response
543
+
asserts the deltas are committed, and acknowledging data that was not stored is
544
+
data loss with better branding. Shadow mode is unaffected: there the legacy raw
545
+
path is still the source of truth.
546
+
452
547
## Security & Supply Chain
453
548
454
549
OtelContext targets the OpenSSF Best Practices `passing` badge (project [12646](https://www.bestpractices.dev/en/projects/12646)) and ships a six-job OSS-CLI security stack, supplemented by **SonarCloud SAST as a required gate** (board reversal 2026-04-28). No CodeQL, no NVD-direct tooling. Cost: $0 for the OSS-CLI tier; SonarCloud is free for public repos.
returnfmt.Errorf("EXEMPLAR_RETENTION_DAYS must be between 1 and HOT_RETENTION_DAYS (%d), got %d: the exemplar tier is a shorter-lived subset of hot retention, never a longer-lived one", c.HotRetentionDays, c.ExemplarRetentionDays)
1088
+
}
1089
+
ifc.ExemplarSynthLogsPerSpan<1 {
1090
+
returnfmt.Errorf("EXEMPLAR_SYNTH_LOGS_PER_SPAN must be >= 1, got %d", c.ExemplarSynthLogsPerSpan)
returnfmt.Errorf("EXEMPLAR_SYNTH_LOGS_PER_TRACE (%d) must be >= EXEMPLAR_SYNTH_LOGS_PER_SPAN (%d): a per-trace cap below the per-span cap makes the per-span cap unreachable", c.ExemplarSynthLogsPerTrace, c.ExemplarSynthLogsPerSpan)
1094
+
}
1095
+
ifc.DataDiskBudgetMB<64 {
1096
+
returnfmt.Errorf("DATA_DISK_BUDGET_MB must be >= 64, got %d", c.DataDiskBudgetMB)
1097
+
}
1098
+
ifstrings.TrimSpace(c.DataDiskPath) =="" {
1099
+
returnfmt.Errorf("DATA_DISK_PATH must not be empty")
1100
+
}
1052
1101
1053
1102
// Sum-of-caps validation: sub-caps must fit under global cap
0 commit comments