Telemetry & observability pipeline with Datadog integration#43
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Introduces the telemetry wire protocol used across all EigenInference
components. Defines a single TelemetryEvent JSON shape with source,
severity, kind, machine_id, request_id, session_id, message, fields,
and stack.
- coordinator/internal/protocol/telemetry.go — canonical Go definition
- provider/src/telemetry/{mod,event,client,queue,layer,panic_hook,stderr_scraper}.rs — Rust mirror + machinery stubs
- console-ui/src/lib/telemetry-types.ts — TS mirror for the browser
- provider: fixed Cargo.toml so non-macOS deps (axum, anyhow, base64, chrono, etc.) compile everywhere, not just macOS
- provider: added once_cell + backtrace deps
- provider: fixed inprocess_engine type-inference issue when building without the python feature
Unit tests cover JSON round-trip, optional-field omission, and the
disk-backed overflow queue. Phase 1 (coordinator ingestion) next.
Co-authored-by: Gajesh Naik <Gajesh2007@users.noreply.github.com>
Adds the full ingestion pipeline on the coordinator side.
Storage
- Postgres: new telemetry_events table with indexes on ts, source+severity,
kind, machine_id, and request_id. Inserts use pgx CopyFrom.
- Memory: bounded 10_000-event ring buffer with the same read API.
- Common: TelemetryEventRecord + TelemetryFilter + TelemetryKindCount types
in the store package.
API
- POST /v1/telemetry/events — batch ingestion from providers, the macOS app,
and the web console. Max 64KB body, max 100 events/batch, per-source
token-bucket rate limiting (200 burst + 100/min authenticated,
30 burst + 10/min anonymous). Fields outside the server allowlist are
silently dropped so free-form user data can never leak into telemetry.
Unknown source/severity/kind values are coerced to safe defaults
(forward-compat with newer clients). Providers are automatically
identified by their device-linked auth token; machine_id is derived
deterministically from a SHA-256 digest.
- GET /v1/admin/telemetry — admin-only list with full filtering.
- GET /v1/admin/telemetry/summary — admin-only rollup by kind.
- GET /v1/admin/metrics — admin-only metrics snapshot (JSON or Prometheus
text format).
Metrics
- New Metrics registry (counters, histograms, computed gauges) with a
stable key scheme, default latency buckets, and Prometheus text encoder.
- Default gauge providers_online registered at boot.
- Ingestion handler bumps telemetry_events_total{source,severity,kind}.
Retention
- New internal/telemetry package. RunRetentionLoop prunes events older than
EIGENINFERENCE_TELEMETRY_MAX_AGE (default 14d) every
EIGENINFERENCE_TELEMETRY_PRUNE_INTERVAL (default 1h). Wired into main.go.
Tests
- Memory store: insert/list/filter/ring-buffer-overflow/prune/count-by-kind.
- HTTP handlers: anon ingest, malformed JSON, oversized batch, oversized
body, empty-message rejection, rate limit, enum coercion, admin auth,
admin list+summary, allowlist sanity, long-message truncation.
Full go test ./... passes across all packages.
Co-authored-by: Gajesh Naik <Gajesh2007@users.noreply.github.com>
Adds panic recovery, in-process metrics wired everywhere, and a
coordinator-side telemetry emitter that writes directly to the store.
Emitter (internal/telemetry/emitter.go)
- NewEmitter(logger, store, metrics, version) — single coordinator-wide
emitter. Persists one row per event, mirrors to slog at the appropriate
level, bumps telemetry_events_total via the minimal MetricsSink adapter.
- Event type exposes severity / kind / message / fields / request_id / stack.
- Per-process SessionID so admin UI can group events from one boot.
Server wiring
- Server holds *telemetry.Emitter + *Metrics. SetEmitter from main.go.
- New helpers emit / emitRequest / emitPanic for call-site ergonomics.
- Metrics() getter so main.go can pass the registry into the emitter.
Recover middleware
- New recoverMiddleware wrapped outside loggingMiddleware; catches panics
in any handler, captures debug.Stack, emits a fatal panic event, and
returns 500. Propagates http.ErrAbortHandler unchanged.
Metrics rollout
- HTTP requests: http_requests_total{method,path,status} +
http_request_duration_ms histogram.
- Inference path: inference_dispatches_total{result=success|retry|timeout|failure}.
- Attestation: attestation_failures_total{reason}.
- Provider lifecycle: provider_registrations_total{trust_level},
ws_disconnects_total{reason}.
- Default gauges: providers_online, pending_image_uploads.
Telemetry event sites in consumer.go / provider.go
- inference retry / first-chunk timeout / dispatch-exhausted failure.
- provider registration (with trust level + chip).
- attestation challenge failure (promoted to error when threshold reached).
- WebSocket read errors.
Prometheus renderer
- Labels are serialized correctly ("k"="v"), le= suffix is merged into
histogram bucket labels instead of concatenated to the metric name.
Tests
- metrics_test.go covers counter add, stable-label keying regardless of
order, histogram bucket cumulative counts + sum, gauge recompute, and
Prom text format.
- recover_middleware_test.go panics a handler and asserts a stored
telemetry event with non-empty stack.
Full go test ./... is green.
Co-authored-by: Gajesh Naik <Gajesh2007@users.noreply.github.com>
…connect events
Wires the Rust provider into the telemetry pipeline.
Boot (cmd_serve)
- Initializes the telemetry client with the coordinator URL, device-linked
auth token, machine_id (SE public key), and version.
- Installs the panic hook BEFORE the subprocess spawns so any crash during
model load / backend bringup is captured with a full backtrace.
- Emits a provider_start event with hardware context (chip, memory).
Backend monitor (backend/mod.rs)
- Health-check failures now emit a backend_crash event with the backend
name and the error (if any). Restart failures emit a second event
so operators see both the trigger and the recovery outcome.
Coordinator client (coordinator.rs)
- Reconnect storms are reported at exponential intervals (attempts 3, 10,
then every 30) so transient Wi-Fi hiccups don't spam the admin console.
- Events include the reconnect count + last error + ws_state.
Tests
- tests/telemetry_symmetry.rs — self-contained Rust fixture asserting the
wire JSON shape (enum casing, optional omission) matches Go.
- tests/telemetry_integration.rs — real reqwest POST against a local axum
stub, confirming the batch shape the client sends is valid.
- coordinator/internal/protocol/telemetry_symmetry_test.go — Go canonical
encoding check.
Also: full cargo test (246 unit + 3 integration) still passes.
Co-authored-by: Gajesh Naik <Gajesh2007@users.noreply.github.com>
Adds a Swift-side telemetry reporter that ships app-level failures to the
coordinator ingest endpoint using the same wire protocol as the provider.
TelemetryReporter
- Singleton reporter with a bounded 500-event in-memory buffer.
- Debounced batch flush (3s idle, max 100 events per batch).
- Server-side allowlist mirrored client-side so we never accidentally
ship prompts or free-form user content.
- Stable per-install machine ID (UUID stored in UserDefaults).
- Per-launch session ID so admin UI can group events from a run.
- Retries on transport error without unbounded growth.
App wiring
- EigenInferenceApp/AppDelegate installs NSSetUncaughtExceptionHandler
that ships a fatal panic event with the full call stack.
- configureTelemetry() resolves the coordinator URL from ConfigManager,
converting wss:// -> https:// so the ingest endpoint reaches the same
host as the WebSocket.
ProviderManager
- Subprocess crashes now emit a structured backend_crash event with
exit_code / signal / attempt / model / truncated last-stderr.
- Startup failures emit a backend_crash event too so operators don't
miss spawn-time problems.
- Restart-limit-exceeded upgrades to severity=fatal with reason
'restart_limit_exceeded'.
Tests
- TelemetryReporterTests covers enum raw values (wire contract with Go),
wss/ws -> https/http URL mapping, machine_id stability, and session_id
presence. Runs in macOS CI where swift is available.
Co-authored-by: Gajesh Naik <Gajesh2007@users.noreply.github.com>
Hooks the Next.js console into the telemetry pipeline.
Client (src/lib/telemetry.ts)
- Debounced batch sender (100 events max per flush, 2s debounce).
- Client-side allowlist mirror so we don't even send prompts / nested
objects that the coordinator would drop.
- Per-session UUID + proper id/timestamp stamping.
- Uses navigator.sendBeacon on page unload when available; falls back
to fetch with keepalive.
- On transport failure, the batch is re-queued oldest-first with a
hard cap so live errors are never starved.
- Exports _resetForTest + _bufferSize for vitest.
Proxy route (src/app/api/telemetry/route.ts)
- POST /api/telemetry forwards to {NEXT_PUBLIC_COORDINATOR_URL}/v1/telemetry/events.
- 64KB body cap matches coordinator's limit.
- Pass-through of Authorization header for account attribution.
- 10s upstream timeout so a wedged coordinator can't stall page unload.
- 502 on upstream failure so the client re-queues correctly.
Global handlers (TelemetryInitializer + global-error.tsx)
- window.error and unhandledrejection forwarders emit http_error events.
- TelemetryInitializer emits a session_start log on first mount.
- Next.js global-error.tsx boundary catches unrecoverable render
errors, reports fatal + renders a friendly fallback with a reset
button. Stack trace included.
Tests (console-ui/__tests__/telemetry.test.ts)
- Happy-path forwarding (202 + correct URL + auth header).
- Oversized body rejection (413).
- Upstream failure returns 502.
- Client-side allowlist filtering.
TS + lint clean on all new files. Pre-existing test failures on master
are untouched.
Co-authored-by: Gajesh Naik <Gajesh2007@users.noreply.github.com>
Adds a /stats/telemetry page to the console so admins can inspect the
live event stream and triage incidents without SSHing into the DB.
Page (console-ui/src/app/stats/telemetry/page.tsx)
- Last-24h rollup tiles grouped by (source, severity, kind).
- Filterable event list (source, severity, kind).
- Per-event expandable details with fields JSON and stack trace.
- Color-coded severity column.
- Relies on coordinator-side admin enforcement: non-admin Privy users
get a 403 and a friendly message rather than a broken page.
Proxy routes
- GET /api/admin/telemetry -> coordinator GET /v1/admin/telemetry.
- GET /api/admin/telemetry/summary -> coordinator rollup endpoint.
- Pass-through of Authorization header so the coordinator is the
single source of truth for access control.
Co-authored-by: Gajesh Naik <Gajesh2007@users.noreply.github.com>
- docs/telemetry.md: full architecture, wire protocol, endpoint table, privacy/allowlist contract, retention config, schema, per-component emission sites, and a 'how to add a new emit site' guide. - AGENTS.md / CLAUDE.md: telemetry symmetry sync points (Go + Rust + TS) and the field-allowlist privacy invariant called out in 'Common Pitfalls'. Co-authored-by: Gajesh Naik <Gajesh2007@users.noreply.github.com>
Adds a coordinator-side E2E test that drives the full ingestion pipeline
through the real HTTP server:
- POST /v1/telemetry/events with an intentionally rogue 'prompt' field.
- Assert admin list requires auth (403 without, 200 with).
- Assert the stored event has the prompt field STRIPPED (privacy
invariant).
- Assert the admin summary endpoint returns counts.
- Assert JSON metrics snapshot contains the ingestion counter.
- Assert Prometheus text format is well-formed.
- Seed a 30-day-old event, run the retention loop, confirm deletion.
Adds telemetry.RunRetentionLoopOnce for synchronous prune trigger, used
by the test above.
Full coordinator suite (go test ./...): all packages green.
Full provider suite (cargo test --no-default-features): 246 unit + 3
integration tests green.
Live binary run (in-memory store) verified:
- 1 event ingest -> 202, allowlist strips 'prompt' field.
- admin list returns cleaned event.
- metrics endpoint shows http_requests_total + telemetry_events_total.
- Prometheus format renders correctly.
- Anon rate limiter: first 20-event flood accepted, next three flushes
rejected with 429.
- Malformed JSON -> 400.
Walkthrough artifact: /opt/cursor/artifacts/telemetry-live-demo.md
Co-authored-by: Gajesh Naik <Gajesh2007@users.noreply.github.com>
CI's gofmt -l check on the Coordinator Tests job flagged three files authored in this PR. Pure whitespace/alignment diffs from gofmt -w; no behavior change. internal/api/server.go internal/api/telemetry_handlers.go internal/api/telemetry_handlers_test.go go test ./... still green. Co-authored-by: Gajesh Naik <Gajesh2007@users.noreply.github.com>
Lint cleanup for the new telemetry handler — the orphan prettyJSON helper was never wired up, and 'max' shadows a Go 1.21+ predeclared builtin. Renamed to maxLen. The remaining Coordinator Lint job failures (gochecknoglobals, sloglint, gocognit, testpackage) match the existing codebase style and are already failing on master, not introduced by this PR. Co-authored-by: Gajesh Naik <Gajesh2007@users.noreply.github.com>
Replace self-hosted Postgres telemetry storage with Datadog integration: - Add dd-trace-go APM for automatic request tracing on coordinator - Add DogStatsD metrics: HTTP latency, inference dispatches, provider registrations, WS disconnects, attestation failures, queue depth - Forward provider/app/console telemetry events to Datadog Logs API via batched async forwarder (5s flush / 100-event batches) - Emit Datadog Events for fatal-severity alerts (monitor-triggerable) - Add slog handler for dd.trace_id/dd.span_id log-trace correlation - Add Datadog RUM to console UI with session replay and user identity - Strip Postgres telemetry queries, admin dashboard, retention loop - Keep provider→coordinator event relay (providers can't run DD agent) - Keep field allowlisting and rate limiting on ingestion endpoint - Fix provider Cargo.toml duplicate macos dependency section - Fix goroutine race in log flush WaitGroup initialization
Gajesh2007
force-pushed
the
cursor/production-system-telemetry-b491
branch
from
April 22, 2026 13:34
7748476 to
34c8202
Compare
Gajesh2007
changed the title
- Replace Math.random() fallback with crypto.getRandomValues for session ID generation in telemetry client - Cap Fields map size before allocation to prevent integer overflow in Datadog log forwarding
Fetch DD_API_KEY and DD_SITE from GCP Secret Manager alongside existing secrets. Sets DD_ENV=development and DD_SERVICE for the dev coordinator on GCE.
Gajesh2007
marked this pull request as ready for review
April 22, 2026 14:00
Gajesh2007
added a commit
that referenced
this pull request
Jun 17, 2026
Picks up the dequant makeMask fast path (MLX #3384 avoidance on the GPT-OSS dequant path) and the composite-CacheList guard in cacheFactoryKind.
Gajesh2007
added a commit
that referenced
this pull request
Jun 18, 2026
- BatchScheduler: reserve VLM media requests at the fp16 KV rate, not the quantized rate. VLM media streams through container.generate (fp16 KV, not the quantized batched cache), so reserving at the ~0.52x quantized rate under-counts ~2x and risks unified-memory OOM under concurrent media. New fp16KVBytesPerToken; no-op when KV quant is off. - kv-attn-selftest: exit(1) when any DAR gate fails (was always exit 0 -> CI would treat a failing correctness gate as success). - kv-engine-demo: enforce --generation-timeout in the concurrency sweep (per-stream timeout race) so a stalled stream aborts instead of hanging; also dump full fp16-vs-quant generations for eyeball quality diffing. - kv-quant-gate: --candidate auto honors DARKBLOOM_KV_GPTOSS_KERNEL=1 (kernel vs dequant), mirroring the live scheduler. - Bump mlx-swift-lm -> quantized-cache IOGPU leak fix (#43). Note: Codex 'kernel silently dequant for Gemma' (Scheduler.swift:947) verified as a FALSE POSITIVE — the served gemma-4-26b checkpoint has vision_config, so it loads via the VLM Gemma4 class which DOES use the quantized kernel (Gemma4.swift:839/893); the update-only LLM Gemma4Text class is never instantiated for it.
Gajesh2007
added a commit
that referenced
this pull request
Jun 20, 2026
- BatchScheduler: reserve VLM media requests at the fp16 KV rate, not the quantized rate. VLM media streams through container.generate (fp16 KV, not the quantized batched cache), so reserving at the ~0.52x quantized rate under-counts ~2x and risks unified-memory OOM under concurrent media. New fp16KVBytesPerToken; no-op when KV quant is off. - kv-attn-selftest: exit(1) when any DAR gate fails (was always exit 0 -> CI would treat a failing correctness gate as success). - kv-engine-demo: enforce --generation-timeout in the concurrency sweep (per-stream timeout race) so a stalled stream aborts instead of hanging; also dump full fp16-vs-quant generations for eyeball quality diffing. - kv-quant-gate: --candidate auto honors DARKBLOOM_KV_GPTOSS_KERNEL=1 (kernel vs dequant), mirroring the live scheduler. - Bump mlx-swift-lm -> quantized-cache IOGPU leak fix (#43). Note: Codex 'kernel silently dequant for Gemma' (Scheduler.swift:947) verified as a FALSE POSITIVE — the served gemma-4-26b checkpoint has vision_config, so it loads via the VLM Gemma4 class which DOES use the quantized kernel (Gemma4.swift:839/893); the update-only LLM Gemma4Text class is never instantiated for it.
Gajesh2007
added a commit
that referenced
this pull request
Jun 20, 2026
…ntation Submodule mlx-swift-lm -> 0f46814 (darkbloom/kvquant, #43): on-GPU penalty sampler, KV-quant SDPA mx.compile + n==1 decode-mask skip, per-token decode wins (logSumExp skip, B==1 .item(), resource-debug off-by-default). kv-engine-demo: add submit->first-token TTFT and prefill tok/s measurement (measurement-only).
Gajesh2007
added a commit
that referenced
this pull request
Jun 20, 2026
mlx-swift-lm #43 squash-merged to main as a9e0ee1 and its branch was deleted, orphaning the prior submodule pointer. Re-point to the merged main commit (identical engine tree to the tested head, no behavior change) so the submodule is fetchable from a clean checkout/CI.
Gajesh2007
added a commit
that referenced
this pull request
Jun 20, 2026
mlx-swift-lm #43 squash-merged to main (a9e0ee1) and its branch was deleted, orphaning the prior pointer. Re-point to the merged main commit so the submodule is fetchable from a clean checkout/CI.
Gajesh2007
added a commit
that referenced
this pull request
Jun 20, 2026
…-313) (#377) * feat(provider): KV quant benchmark gate with V-only affine4 pass - Add kv-quant-gate CLI and supporting Benchmark/KVQuant modules for PPL, logits, output, NIAH, performance, and memory suites. - Implement custom KVQuantizedCache and VOnlyQuantizedKVCache wrappers with start-token delay and sliding-window preservation. - Wire candidate cache injection into the performance runner so reference and candidate metrics are collected end-to-end. - Resolve HF-cache symlinks before file-extension checks in LocalMLXModelReadiness so downloaded snapshots load correctly. - Shift release thresholds to agreement metrics where the fp16 baseline itself can fail (output/NIAH), and add json_valid/required_substrings agreement rates. - Default candidate is full-v-affine4:g64:start1024; affine4/8 full-KV and turbo modes are currently rejected as unsupported by the gate. V-only affine4 passes quality gates for both Gemma 4 26B and GPT-OSS-20B. * feat(provider): bfloat16 KV cache mode and benchmark-gated quality - Add `bf16-kv:start1024` candidate mode with a custom BFloat16KVCache wrapper. This halves KV-cache precision with no model-attention changes and passes PPL/logits/output/NIAH gates for both Gemma 4 26B and GPT-OSS-20B. - Switch `affine4/8` full-KV modes to mlx-swift-lm upstream dynamic quantization (GenerateParameters.kvBits) and add an incremental teacher-forced scorer so the PPL/logits suites can exercise the same quantized generation path. (Upstream QuantizedKVCache fatal-errors on the single-forward update path, so the old scorer could not be used.) - Reset MLX peak memory between reference and candidate baseline runs so peak-memory ratios are per-run rather than cumulative. - Use peak-memory max (instead of mean) for threshold memory ratios. - Update KVQuantExecutionTests for the new affine4/8/bf16 behavior. * feat(provider): add full-v-bf16 KV cache candidate - Add `full-v-bf16:start1024` candidate mode and VOnlyBFloat16KVCache wrapper: keys stay fp16, values are stored as bfloat16 after the start token. This is intended as a middle-ground compression option that works with model attentions using the plain update(keys:values:) path. - Quality gate passes for Gemma 4 26B; memory/perf impact is still being measured against the current total-GPU-peak metric. - Update KVQuantExecutionTests for the new mode. * feat(provider): protocol-safe quantized cache wrapper and policy alignment - Add ProtocolSafeQuantizedKVCache: wraps upstream QuantizedKVCache and implements update(keys:values:) with a dequantizing fallback. This lets full-KV affine modes run through both the native quantized attention path and the plain update path used by single-forward scoring. - Switch affine4/8 execution back to a custom cache factory (the protocol-safe wrapper) so the gate can evaluate full-KV quantization end-to-end. - Add a minimum PPL ratio threshold (0.99) so large candidate-perplexity decreases are also flagged as failures. - Update KVQuantPolicy to recommend the validated `full-v-affine4:g64:start1024` candidate for Gemma 4 and GPT-OSS, and adjust KVQuantPolicyTests. * feat(kvquant): prove kernel correctness, fix scorer measurement bug, add capacity metric First-principles instrumentation for the KV-cache capacity objective: - kv-attn-selftest executable: deterministic numerical probes proving quantizedScaledDotProductAttention == dequantize+SDPA exactly (MHA/GQA/ causal/decode, 8/4-bit), so the kernel is correct after the mask-fill fix. Outlier + multi-step + per-channel isolation cases included. - Fix mask-fill bug in quantizedScaledDotProductAttention (vendored): masked positions used +leastNormalMagnitude (~0) instead of -inf. - Fix a measurement bug: the single-forward PPL/logits scorer let start-delayed caches silently return fp16 (never engaging quantization). Candidate scoring now runs true incremental token-by-token through the candidate's own cache. KVQuantCacheProbe deterministically validates single-forward vs incremental divergence (one-shot diff 0.0 vs incremental 0.31) — no 26B run needed. - Add a model-free capacity suite + per-mode stored K/V bits and capacityRatioVsFP16 (incl. affine scale/bias overhead). Exposes that bf16 modes are 1.0x (no capacity gain) and affine8 is ~1.78x (overhead), making KV-bytes-per-token the headline metric. Finding: Gemma 4 K8V8 collapse (top1 0.56) is genuine compounded quantization loss across ~48 layers, NOT a kernel bug (dequant path == kernel path == 0.5595). Per-layer cos 0.999 still compounds to large end-to-end drift. * feat(kvquant): trustworthy generation-fidelity gate; honest quality/capacity Pareto Raw-text teacher-forced PPL/logits was an untrustworthy gauge (reference NLL ~5.3 nats / PPL ~200 on trivial text; ppl.ratio<1). Replace the logits suite with generation-fidelity: the reference greedily continues a coherent prompt, the candidate is teacher-forced over that continuation, and we measure top-1 / top-5 agreement on confident tokens through the correct generation forward (quantization engaged incrementally). reference-self match = 1.000 validates it. Honest Gemma 4 26B Pareto (capacity includes affine scale/bias overhead): V-only 4-bit start1024 : 1.56x, top1 1.000 K8V8 start0 : 1.78x, top1 0.931, top5 0.993 K4V4 start0 : 3.56x, top1 0.892, top5 0.941 Conclusion: earlier 'catastrophic' 0.56 numbers were gauge artifacts, not real quantization loss. V4 is effectively lossless; K8V8 is already 0.93 with no start delay. Next: long-context stress + start-delay/residual window + asymmetric K8/V4 to reach quality-preserving 2x. * feat(kvquant): establish quality-preserving ~2x via long-context gen-fidelity sweep Long-context generation-fidelity (2048-3072 tok prompts, ~768 samples) gives a trustworthy quality x capacity Pareto on BOTH target models: K8V8 g128 -> 1.94x : Gemma 0.973 top1, GPT-OSS 0.982 top1 (robust ~2x) K6V6 g64 -> 2.46x : Gemma 0.977 top1, GPT-OSS 0.932 (min 0.78, riskier) V4 -> 1.56x : ~0.995 (near-lossless anchor) Path selection (both store quantized = capacity gain): - Gemma 4: kernel path (ProtocolSafeQuantizedKVCache, quantized attention). - GPT-OSS: kernel path FATALS ('Quantized attention does not support non-zero sinks'); use dequant path (KVQuantizedCache) which is sink-safe and still stores quantized. Gate plumbing: long coherent prompts truncated to target context; prompt primed in one batched forward then decoded token-by-token; KVQ_BITS/KVQ_GROUP/KVQ_DEQUANT env knobs for Pareto sweeps. Conclusion: ~2x capacity at high fidelity is real; K8V8 g128 is the conservative cross-model pick. * DAR-313: make validated KV-quant configs first-class modes, remove env-var sweep hacks - Remove KVQ_BITS/KVQ_GROUP/KVQ_DEQUANT env reads from affine4/affine8 execution. - Add first-class modes: k8v8:g128, k8v8:g64:dequant, k6v6:g64, k6v6:g64:dequant. - Wire all KVQuantCandidateMode switches (types, cacheSpec, execution config). - Update effectiveBits to use the mode's actual group size for accurate capacity ratio. - Add parse + factory-non-nil tests for the four new modes. * chore(submodule): bump mlx-swift-lm to darkbloom/kvquant (KV-quant mask fix) [DAR-313] * chore(kvquant): point engine submodule at merged main (#43 -> a9e0ee1) mlx-swift-lm #43 squash-merged to main (a9e0ee1) and its branch was deleted, orphaning the prior pointer. Re-point to the merged main commit so the submodule is fetchable from a clean checkout/CI.
Gajesh2007
added a commit
that referenced
this pull request
Jun 20, 2026
) * feat(provider): KV quant benchmark gate with V-only affine4 pass - Add kv-quant-gate CLI and supporting Benchmark/KVQuant modules for PPL, logits, output, NIAH, performance, and memory suites. - Implement custom KVQuantizedCache and VOnlyQuantizedKVCache wrappers with start-token delay and sliding-window preservation. - Wire candidate cache injection into the performance runner so reference and candidate metrics are collected end-to-end. - Resolve HF-cache symlinks before file-extension checks in LocalMLXModelReadiness so downloaded snapshots load correctly. - Shift release thresholds to agreement metrics where the fp16 baseline itself can fail (output/NIAH), and add json_valid/required_substrings agreement rates. - Default candidate is full-v-affine4:g64:start1024; affine4/8 full-KV and turbo modes are currently rejected as unsupported by the gate. V-only affine4 passes quality gates for both Gemma 4 26B and GPT-OSS-20B. * feat(provider): bfloat16 KV cache mode and benchmark-gated quality - Add `bf16-kv:start1024` candidate mode with a custom BFloat16KVCache wrapper. This halves KV-cache precision with no model-attention changes and passes PPL/logits/output/NIAH gates for both Gemma 4 26B and GPT-OSS-20B. - Switch `affine4/8` full-KV modes to mlx-swift-lm upstream dynamic quantization (GenerateParameters.kvBits) and add an incremental teacher-forced scorer so the PPL/logits suites can exercise the same quantized generation path. (Upstream QuantizedKVCache fatal-errors on the single-forward update path, so the old scorer could not be used.) - Reset MLX peak memory between reference and candidate baseline runs so peak-memory ratios are per-run rather than cumulative. - Use peak-memory max (instead of mean) for threshold memory ratios. - Update KVQuantExecutionTests for the new affine4/8/bf16 behavior. * feat(provider): add full-v-bf16 KV cache candidate - Add `full-v-bf16:start1024` candidate mode and VOnlyBFloat16KVCache wrapper: keys stay fp16, values are stored as bfloat16 after the start token. This is intended as a middle-ground compression option that works with model attentions using the plain update(keys:values:) path. - Quality gate passes for Gemma 4 26B; memory/perf impact is still being measured against the current total-GPU-peak metric. - Update KVQuantExecutionTests for the new mode. * feat(provider): protocol-safe quantized cache wrapper and policy alignment - Add ProtocolSafeQuantizedKVCache: wraps upstream QuantizedKVCache and implements update(keys:values:) with a dequantizing fallback. This lets full-KV affine modes run through both the native quantized attention path and the plain update path used by single-forward scoring. - Switch affine4/8 execution back to a custom cache factory (the protocol-safe wrapper) so the gate can evaluate full-KV quantization end-to-end. - Add a minimum PPL ratio threshold (0.99) so large candidate-perplexity decreases are also flagged as failures. - Update KVQuantPolicy to recommend the validated `full-v-affine4:g64:start1024` candidate for Gemma 4 and GPT-OSS, and adjust KVQuantPolicyTests. * feat(kvquant): prove kernel correctness, fix scorer measurement bug, add capacity metric First-principles instrumentation for the KV-cache capacity objective: - kv-attn-selftest executable: deterministic numerical probes proving quantizedScaledDotProductAttention == dequantize+SDPA exactly (MHA/GQA/ causal/decode, 8/4-bit), so the kernel is correct after the mask-fill fix. Outlier + multi-step + per-channel isolation cases included. - Fix mask-fill bug in quantizedScaledDotProductAttention (vendored): masked positions used +leastNormalMagnitude (~0) instead of -inf. - Fix a measurement bug: the single-forward PPL/logits scorer let start-delayed caches silently return fp16 (never engaging quantization). Candidate scoring now runs true incremental token-by-token through the candidate's own cache. KVQuantCacheProbe deterministically validates single-forward vs incremental divergence (one-shot diff 0.0 vs incremental 0.31) — no 26B run needed. - Add a model-free capacity suite + per-mode stored K/V bits and capacityRatioVsFP16 (incl. affine scale/bias overhead). Exposes that bf16 modes are 1.0x (no capacity gain) and affine8 is ~1.78x (overhead), making KV-bytes-per-token the headline metric. Finding: Gemma 4 K8V8 collapse (top1 0.56) is genuine compounded quantization loss across ~48 layers, NOT a kernel bug (dequant path == kernel path == 0.5595). Per-layer cos 0.999 still compounds to large end-to-end drift. * feat(kvquant): trustworthy generation-fidelity gate; honest quality/capacity Pareto Raw-text teacher-forced PPL/logits was an untrustworthy gauge (reference NLL ~5.3 nats / PPL ~200 on trivial text; ppl.ratio<1). Replace the logits suite with generation-fidelity: the reference greedily continues a coherent prompt, the candidate is teacher-forced over that continuation, and we measure top-1 / top-5 agreement on confident tokens through the correct generation forward (quantization engaged incrementally). reference-self match = 1.000 validates it. Honest Gemma 4 26B Pareto (capacity includes affine scale/bias overhead): V-only 4-bit start1024 : 1.56x, top1 1.000 K8V8 start0 : 1.78x, top1 0.931, top5 0.993 K4V4 start0 : 3.56x, top1 0.892, top5 0.941 Conclusion: earlier 'catastrophic' 0.56 numbers were gauge artifacts, not real quantization loss. V4 is effectively lossless; K8V8 is already 0.93 with no start delay. Next: long-context stress + start-delay/residual window + asymmetric K8/V4 to reach quality-preserving 2x. * feat(kvquant): establish quality-preserving ~2x via long-context gen-fidelity sweep Long-context generation-fidelity (2048-3072 tok prompts, ~768 samples) gives a trustworthy quality x capacity Pareto on BOTH target models: K8V8 g128 -> 1.94x : Gemma 0.973 top1, GPT-OSS 0.982 top1 (robust ~2x) K6V6 g64 -> 2.46x : Gemma 0.977 top1, GPT-OSS 0.932 (min 0.78, riskier) V4 -> 1.56x : ~0.995 (near-lossless anchor) Path selection (both store quantized = capacity gain): - Gemma 4: kernel path (ProtocolSafeQuantizedKVCache, quantized attention). - GPT-OSS: kernel path FATALS ('Quantized attention does not support non-zero sinks'); use dequant path (KVQuantizedCache) which is sink-safe and still stores quantized. Gate plumbing: long coherent prompts truncated to target context; prompt primed in one batched forward then decoded token-by-token; KVQ_BITS/KVQ_GROUP/KVQ_DEQUANT env knobs for Pareto sweeps. Conclusion: ~2x capacity at high fidelity is real; K8V8 g128 is the conservative cross-model pick. * DAR-313: make validated KV-quant configs first-class modes, remove env-var sweep hacks - Remove KVQ_BITS/KVQ_GROUP/KVQ_DEQUANT env reads from affine4/affine8 execution. - Add first-class modes: k8v8:g128, k8v8:g64:dequant, k6v6:g64, k6v6:g64:dequant. - Wire all KVQuantCandidateMode switches (types, cacheSpec, execution config). - Update effectiveBits to use the mode's actual group size for accurate capacity ratio. - Add parse + factory-non-nil tests for the four new modes. * chore(submodule): bump mlx-swift-lm to darkbloom/kvquant (KV-quant mask fix) [DAR-313] * feat(kvquant): batched quantized-cache numerical gate + submodule bump (DAR-314) Adds QuantizedBatchKVCache deterministic correctness gate to kv-attn-selftest (single-row/left-padding/GQA/growth + finalize/extend/filter, dequant-same-data reference) and bumps mlx-swift-lm to the QuantizedBatchKVCache commit. * feat(provider): wire K8V8 KV-quant into batching engine, off by default (DAR-317/318) Opt-in backend.kv_quant flag (default off); Gemma-4-only via KVQuantPolicy. Quantized full-attention BatchKVCache (g128/b8), fp16 sliding; KV byte budget cut ~1.94x so admission grants ~2x tokens; prefix-cache/MTP disabled when on. Flag-off verified byte-for-byte no-regression. Bumps mlx-swift-lm pointer. * feat(provider): kv-engine-demo (e2e capacity proof) + --local flag + policy doc (DAR-318) kv-engine-demo builds the real BatchedEngine fp16 vs K8V8 g128 on Gemma 4: capacity kv_bytes/token 0.516x -> token budget 1.95x; word-level output parity (sanity; rigorous quality = single-seq top1 0.973); decode tok/s ~parity. StartCommand --local now honors backend.kv_quant; KVQuantPolicy doc reflects k8v8:g128. Read-only telemetry accessors for the harness. * feat(provider): GPT-OSS KV-quant on continuous batching via dequant cache (DAR-322) resolveKVQuantScheme: Gemma -> kernel g128; GPT-OSS -> dequant g64 (head_dim guard); flag default off. Byte budget ~1.88x for GPT-OSS. e2e (real BatchedEngine): capacity 1.88x, rigorous top1 0.971, decode ~0.85x (stable to 512 tok). Bumps submodule pointer; demo label derives from resolved scheme. * feat(kv-engine-demo): long-context decode sweep + dequant microbench (DAR-323) Adds --prompt-tokens sweep, --dequant-iters microbench, --capacity-only. Used to show GPT-OSS dequant decode is a FLAT ~10-15% (not O(n^2)): quant/fp16 ratio 0.86-0.92 across 512..16384; engine overlaps ~85-90% of raw dequant. * fix(kvquant): gate live K+V schemes by capacity * fix(kvquant): address Codex review (#378) — diagnostics, tests, gate - kv-engine-demo: label the quantized run by the actual resolved scheme (Gemma g128 / GPT-OSS dequant|kernel / fp16 for unsupported) and reject unsupported model families up front instead of reporting a bogus fp16-vs-fp16 'quantized' comparison. - kv-engine-demo: race stream consumption against --generation-timeout so a stall before/between chunks aborts instead of hanging forever. - KVQuantPerformanceRunner: only apply the decode-TPS regression gate to the performance suite; the memory suite shares the runner and must not fail on throughput. - kv-attn-selftest: fold the live g128 finalize/extend/filter checks into the DAR-314 gate (dar314Ok) instead of a followUpOk that is re-declared later and discarded their failures. - KVQuantEngineTests: mark the suite .serialized; tests mutate the process-wide DARKBLOOM_KV_GPTOSS_KERNEL env and raced under parallel execution. * fix(kvquant): address Codex round-2 review (#378) + submodule leak bump - BatchScheduler: reserve VLM media requests at the fp16 KV rate, not the quantized rate. VLM media streams through container.generate (fp16 KV, not the quantized batched cache), so reserving at the ~0.52x quantized rate under-counts ~2x and risks unified-memory OOM under concurrent media. New fp16KVBytesPerToken; no-op when KV quant is off. - kv-attn-selftest: exit(1) when any DAR gate fails (was always exit 0 -> CI would treat a failing correctness gate as success). - kv-engine-demo: enforce --generation-timeout in the concurrency sweep (per-stream timeout race) so a stalled stream aborts instead of hanging; also dump full fp16-vs-quant generations for eyeball quality diffing. - kv-quant-gate: --candidate auto honors DARKBLOOM_KV_GPTOSS_KERNEL=1 (kernel vs dequant), mirroring the live scheduler. - Bump mlx-swift-lm -> quantized-cache IOGPU leak fix (#43). Note: Codex 'kernel silently dequant for Gemma' (Scheduler.swift:947) verified as a FALSE POSITIVE — the served gemma-4-26b checkpoint has vision_config, so it loads via the VLM Gemma4 class which DOES use the quantized kernel (Gemma4.swift:839/893); the update-only LLM Gemma4Text class is never instantiated for it. * fix(kvquant): gate enforcement + Gemma head_dim guard + selftest exit; bump engine Provider/gate (safe-set review fixes): - kv-quant-gate: exit non-zero when the report is failed; skipped is non-zero unless --allow-missing-data (wires the previously-unused flag). CI can now enforce the gate without parsing JSON. - KVQuantThresholdEvaluator: fail closed on unreadable/malformed threshold JSON, empty rule sets, and zero matched metrics (was silently ignored) + unit tests. - resolveKVQuantScheme: guard Gemma 4 on (global_head_dim ?? head_dim) % 128 == 0 so a malformed/variant config disables KV quant instead of trapping the quantized cache precondition at decode + unit tests. - kv-attn-selftest: fold runCase/runMultiStepCacheTest verdicts into the exit status (early probes could print [FAIL] but still exit 0). - Bump libs/mlx-swift-lm to the engine commit that restricts KV-quant to plain KVCacheSimple and forces the cold path under quantization. Deferred (separate pass): GPT-OSS dequant peak-memory accounting, hybrid sliding-window fp16 reservation, VLM-reservation -> coordinator capacity, and release.json metric emission/pruning. * perf(provider): keep engine concurrency cap in sync with effective cap TASK A — concurrency-cap sync (throughput at concurrency). The continuous-batching engine's per-step concurrency is set by the scheduler via engine.setMaxNumSeqs(...). It was told the cold-start cap exactly ONCE at load and never heard the adaptive ramp: - load pinned dynamicMaxConcurrentRequests to min(4, maxConcurrentRequests) and pushed that single value to the engine; - AdaptiveBatchCapPolicy later raised dynamicMaxConcurrentRequests but only mirrored to the engine when the *adaptive* value changed, and the cold-start burst has no TPS samples yet so the ramp hadn't engaged. Net effect: with maxConcurrentRequests=8 (memory admitting) the engine stayed pinned at 4, so 8 concurrent requests ran as two serialized waves of 4 — roughly halving aggregate throughput at concurrency. Fix: - Add syncEngineConcurrency(): computes the effective cap IDENTICALLY to effectiveMaxConcurrentRequests — max(1, min(maxConcurrentRequests, dynamicMaxConcurrentRequests, memoryBoundMaxConcurrentRequests)) — and pushes it to the engine only when it changed from lastPushedMaxNumSeqs. It is the single setMaxNumSeqs push point. Called at load (replacing the raw setMaxNumSeqs) and on every finish via updateDynamicMaxConcurrentRequests (now re-syncs unconditionally, so a moved memory clamp is honored even when the adaptive cap is unchanged). - Cold-start seed: seed dynamicMaxConcurrentRequests to the configured ceiling max(1, maxConcurrentRequests) at all three sites (init, applyPostLoadBudgets, stopCurrentEngine) instead of min(4, ...). The memory clamp is authoritative in effectiveMaxConcurrentRequests / syncEngineConcurrency, so the engine is NEVER told more than memoryBoundMaxConcurrentRequests (OOM safe) or maxConcurrentRequests. Seeding at the ceiling (rather than min(ceiling, memoryBound)) also lets the effective cap track a RISING memory bound immediately instead of waiting for the slow adaptive ramp. (memoryBoundMaxConcurrentRequests is file-private to the telemetry extension and not referenceable from the seed sites, which is also why the seed defers the clamp to sync.) - Reset lastPushedMaxNumSeqs = -1 in stopCurrentEngine so a freshly-built engine (which starts at its own config.maxNumSeqs default) is always re-pushed on the next load. Adds BatchSchedulerConcurrencyCapTests (non-live, no GPU): with maxConcurrentRequests=8 and memory admitting, the effective cap and the value syncEngineConcurrency pushes are both 8; with bridges that raise the average reservation the memory clamp holds the pushed value at 4 (OOM safety); sync is idempotent; a 1-slot config stays at 1. TASK B — demo throughput metric. kv-engine-demo's concurrency sweep reported aggregate decode tok/s as total decode tokens / the GLOBAL first-token→last-token wall window, which includes later waves' queue-wait/prefill and so under-reports steady-state decode. runConcurrentBatch now returns two clearly-labeled numbers: steadyStateDecodeTps (sum of per-stream decode rates, each over that stream's own decode window) as the capacity headline, and wallClockDecodeTps (the previous total/global-window number) for contrast. Measurement-only; no engine/scheduler behavior change. * perf(provider): bump engine to perf line + TTFT/prefill demo instrumentation Submodule mlx-swift-lm -> 0f46814 (darkbloom/kvquant, #43): on-GPU penalty sampler, KV-quant SDPA mx.compile + n==1 decode-mask skip, per-token decode wins (logSumExp skip, B==1 .item(), resource-debug off-by-default). kv-engine-demo: add submit->first-token TTFT and prefill tok/s measurement (measurement-only). * feat(kvquant): compose KV-quant + prefix cache for the dequant scheme (DAR-319 v2) Relax the mutual-exclusion gate so GPT-OSS (.dequant) gets BOTH ~2x KV capacity AND prefix-cache TTFT. The engine checkpoint-restore now rebuilds QUANTIZED batched caches (mlx-swift-lm 48866bc, restoredFullAttentionCache), so restored rows are concrete-class-compatible with quantized cold rows. The native-kernel scheme (Gemma g128) stays excluded pending its own composition; drafter-MTP unchanged. Submodule mlx-swift-lm -> 48866bc. * chore(kvquant): clean ticket refs from comments + bump engine Remove internal tracker numbers from KV-quant comments and self-test/demo labels for readability (no logic change). Bump mlx-swift-lm to pick up the finish-time prompt-cache extract optimization. * feat(kvquant): document kv_quant opt-in switch + log activation at load Fix the stale BackendSettings.kvQuant doc (KV-quant supports GPT-OSS AND Gemma 4, not Gemma-only) and document the per-provider opt-in. Add a kv-quant logger that, at model load, records whether kv_quant actually engaged for the model (scheme + KV bytes/token + prefix-cache state) or fell back to fp16 — so a beta provider can confirm the switch via 'darkbloom logs'. * chore(kvquant): point engine submodule at merged main (#43 -> a9e0ee1) mlx-swift-lm #43 squash-merged to main as a9e0ee1 and its branch was deleted, orphaning the prior submodule pointer. Re-point to the merged main commit (identical engine tree to the tested head, no behavior change) so the submodule is fetchable from a clean checkout/CI. * chore(kvquant): point engine submodule at merged main (#46 -> 8a9bc7c) Picks up #46 (quantize sliding-window layers: QuantizedBatchRotatingKVCache / DequantBatchRotatingKVCache + windowSize front-trim + sink-safe GPT-OSS kernel) plus the rotating analog of restoredFullAttentionCache so KV-quant composes with prefix-cache restore on sliding-window layers. Engine main 8a9bc7c == previously tested integration (byte-identical tree).
cursor Bot
pushed a commit
that referenced
this pull request
Jul 10, 2026
Admin recover-undispatched now enqueue_critical(inference.released) so refunds cannot silently lose their durable side effect. release_sql gates the same outbox insert on mark+credit. Quiescence stays blocked until drain.
cursor Bot
pushed a commit
that referenced
this pull request
Jul 10, 2026
Wire prepare-fail, prepare-timeout, resize-authorize failure, and pre-start cancel through release_job_with_outbox / enqueue_released so all refunds emit critical outbox side effects (DECISIONS #43).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Production observability pipeline for all components (coordinator, provider, macOS app, console UI) with Datadog as the primary backend.
/v1/telemetry/eventsendpoint, which forwards to Datadog Logs API (5s flush / 100-event batches)dd.trace_idanddd.span_idinto every coordinator log entryArchitecture
What was stripped from the original telemetry PR
/stats/telemetry— use Datadog Log Explorer + DashboardsEnv vars required
Coordinator:
DD_API_KEY— Datadog API keyDD_SITE— Datadog region (e.g.datadoghq.com)DD_ENV—productionordevelopmentDD_DOGSTATSD_URL— DogStatsD address (defaultlocalhost:8125)Console UI (Vercel):
NEXT_PUBLIC_DD_APPLICATION_ID— Datadog RUM application IDNEXT_PUBLIC_DD_CLIENT_TOKEN— Datadog RUM client tokenNEXT_PUBLIC_DD_SITE— Datadog regionSuggested Datadog monitors
logs("service:d-inference-coordinator source:provider severity:fatal").count() > 0p95:trace.http.request{service:d-inference-coordinator} > 10ssum:d_inference.ws.disconnects{*}.as_count() > 50in 15minavg:d_inference.request_queue.depth{*} > 20Test plan
go test ./...— all coordinator packages passcargo test— 281/281 provider tests pass (1 pre-existing flaky under parallel execution)npm run build— console UI builds cleannpm test— 77/78 pass (1 pre-existing unrelated failure in api.test.ts)🤖 Generated with Claude Code