feat(exemplar): disk budget arithmetic inside 8 GiB

[aksOps]

Implements the frozen contract from #201 (map #195, parent #194). Branched from origin/main at 8f0f524 (post-#210).

Q1 — 8 GiB budget table

Tier	Allocation	Covers
Main relational tier	4.5 GiB	Raw trace/span/log exemplars, synthesized logs, investigations and other main-DB metadata, indexes, FTS5, free pages
aggregate.db	1.5 GiB	Buckets, delta log, baselines, identity tables, indexes
DLQ	0.5 GiB	Existing DLQ_MAX_DISK_MB cap
WAL/SHM + temp	0.5 GiB	-wal/-shm sidecars of both DBs, SQLite temp, TLS, transient maintenance
Headroom	1 GiB	Mandatory, unused

Documented in CLAUDE.md (no separate budget doc exists). Every component gets a measured high-water gauge: otelcontext_disk_component_bytes{component} and otelcontext_disk_component_high_water_bytes{component} for main_db|aggregate_db|dlq|wal. Unused allocation in one tier never authorizes another to consume the final 1 GiB — the tiers are separate gauges, not a shared pool.

Q2 — Exemplar retention + window budget

• EXEMPLAR_BYTES_GLOBAL_WINDOW default 8 MiB → 3 MiB. 4 MiB remains configurable, is not the default. Per-service stays 512 KiB.
• New EXEMPLAR_RETENTION_DAYS=2, validated 1..HOT_RETENTION_DAYS. Drives Repository.PurgeExemplarsBatched: exemplar traces + the spans that batch orphans inside one transaction per batch, then logs (FTS rows follow via the content-linked AFTER DELETE trigger — verified in test), then a sweep of expired weak references left by a cancelled pass. Runs on the existing purge tick ahead of the 7-day purge. Aggregate retention unchanged at 7 days.
• Wired only in AGGREGATE_MODE=aggregate. Judgment call: in legacy/shadow the raw rows are the dataset, so a 2-day purge would be data loss rather than budget enforcement.

Q3 — Synthesized-log metering

AllowSynthesizedLog (severity floor only) is replaced by ReserveSynthesizedLog, which reserves len(body)+len(attributesJSON)+logRowFixedBytes against the selected trace's per-trace budget AND the shared per-service/global window budgets, under new EXEMPLAR_SYNTH_LOGS_PER_SPAN=8 / EXEMPLAR_SYNTH_LOGS_PER_TRACE=64. Refusals drop the log, increment otelcontext_exemplar_dropped_total{signal="logs",reason} with synth_per_span|synth_per_trace|budget_bytes, and stamp the trace truncated. They do not consume the ordinary log-exemplar quota. A cheap SynthesizedLogEligible pre-gate keeps INFO span events from paying for a JSON marshal before refusal.

Q4 — Reservation lifecycle

ChargeSpan/AdmitLog → ReserveSpan/ReserveLog taking an *ExemplarReservation. Reserve before row construction → commit when the primary queue or DLQ accepts the batch → release only when the row never reached a destination (dropped pre-submission, or both destinations refused). Reserved bytes bind the cap exactly like committed ones. Once accepted the charge is monotonic for that window: eviction releases the count slot and never the bytes. Batch.Reservation carries the charge to the submit boundary; submitExemplars settles it.

Q5 — Disk watchdog

internal/storage/disk_watchdog.go. statfs on DATA_DISK_PATH is the enforcement source (build-tagged shims for linux/darwin/other, stdlib syscall, no new dependency); ceiling = min(DATA_DISK_BUDGET_MB, usable volume capacity). ≥90% → errors_only (error exemplars only; healthy/slow/WARN off). ≥95% → raw_off (all raw admission off, exemplar DLQ fallback closed, immediate expired-exemplar purge + wal_checkpoint(TRUNCATE), /ready → 503). Hysteresis: leave raw_off below 90%, errors_only below 85%. A failed statfs holds the current state rather than shedding on a syscall error.

Raw shedding never converts a successful aggregate Export into a retryable failure. The exception: aggregate.IsDiskFull classifies ENOSPC/EDQUOT/SQLITE_FULL on the authoritative commit path and maps it to RESOURCE_EXHAUSTED (429 over HTTP OTLP), so the Export fails and the client retries. Shadow mode still swallows it — there the legacy raw path is the source of truth.

Test results (real)

go build ./...                                  ok
go vet ./...                                    ok
go test -race -count=1 ./internal/ingest/... ./internal/storage/... ./internal/aggregate/...
                                                798 passed, 3 packages
go test -count=1 ./...                          1358 passed, 29 packages
golangci-lint run (touched packages)            no new findings

New tests: reservation commit/release/idempotence/merge, reserved bytes binding the cap, committed bytes surviving post-acceptance eviction with the count slot still released, span-slot vs synthesized-log release, Export-boundary commit and release; synthesized-log per-span/per-trace/byte caps with truncation stamping and quota isolation; shedding ladder + hysteresis transition table over a fake statfs, ceiling clamping, stat-failure hold, high-water marks; exemplar purge (2-day gone incl. FTS, hot-tier intact, weak-reference sweep, transactional rollback, scheduler wiring, on-demand trigger); IsDiskFull classification incl. adjacent non-disk-full errors; ENOSPC/SQLITE_FULL → Export failure at both applyAggregate and TraceServer.Export; readiness disk probe; config validation for all four new knobs.

Judgment calls

1. Enforcement is pure statfs, not summed component file sizes. Per the contract, file sizes are attribution gauges only.
2. Exemplar purge is aggregate-mode-only (see Q2 above).
3. "Expired weak references" has no dedicated table in this codebase; the closest real referent is spans whose trace row is gone (the existing PurgeTracesBatched orphan sweep). Implemented as that, bounded to start_time < cutoff so clock-skewed spans under a live trace are never swept.
4. Eviction releases the count slot only. Reserved bytes belonging to an evicted trace stay with the reservation, which commits or releases them at the submit boundary — a second refund path would double-count.
5. ENOSPC maps to RESOURCE_EXHAUSTED, not UNAVAILABLE, so the existing HTTP OTLP mapping produces a retryable 429.
6. New metrics use the otelcontext_ prefix, matching the OTLP metrics completeness scope for aggregate mode #199/Aggregate metadata lifecycle: dictionary GC and miner-state persistence #200 surface rather than the older OtelContext_ gauges.
7. No new SQLite per-driver default flips. The contract did not ask for one and the 3 MiB window is already the SQLite-survival number.

Unverified

The 4.5/1.5/0.5/0.5/1 GiB split is arithmetic and policy; only the seven-day gate (#202) can confirm the real amplification factor. The 2x DB/index/FTS amplification is the contract's provisional assumption, carried into the docs as provisional. Nothing here was run against a real 8 GiB volume under load — the watchdog is exercised through an injected statfs.

[sonarqubecloud]

Quality Gate passed

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.5% Duplication on New Code

See analysis details on SonarQube Cloud