perf(telemetry): worker_threads decode pool (on by default, adaptive) + streamed ClickHouse insert bodies

[simlarsen]

What

Two coupled efficiency changes for the telemetry ingest path:

1. A worker_threads decode pool moves the worker's heaviest synchronous CPU — gunzip + protobuf decode + toJSON() — off the single event loop, so one pod uses multiple cores for decode instead of interleaving jobs on one thread. Enabled by default with adaptive sizing: unset TELEMETRY_DECODE_THREADS resolves to clamp(effectiveCPUs − 1, 0, 4), where effective CPUs honor the pod's cgroup quota (v2 cpu.max, v1 cfs_quota_us, fallback os.availableParallelism). A 1-CPU pod stays at 0 threads — inline decode, byte-for-byte the historical path, no throttling regression. Explicit values always win; 0 is a hard kill switch.
2. Streamed ClickHouse insert bodies (default on; CLICKHOUSE_STREAMED_INSERTS=false to disable): insertJsonRows hands the client a fresh per-invocation object-mode Readable instead of a rows array, so the client encodes incrementally with socket/gzip backpressure instead of materializing the entire body as one string (~100MB+ transient per 100k-row flush) in one long stringify burst. Wire bytes are byte-identical (same encodeJSON path in the client — proven in tests against the vendored encoder). Covers both the worker-direct path and the telemetry-writer tier.

Design highlights

• Pure decode module (OtelDecode.ts) with an infra-free import closure (verified by require-cache dump) shared by the inline path and the thread entry — one implementation, two call sites.
• Resilient pool (DecodeThreadPool.ts): FIFO, one in-flight per thread, copy-then-transfer ArrayBuffer handoff (ioredis reply Buffers are slab views; the copy respects byteOffset/length, pinned by a slab-view regression test), thread-death → retryable rejection + respawn, circuit breaker (5 failures/30s → 60s cooldown) degrading to inline decode, shutdown that settles every pending promise.
• Spawning: App runs via ts-node in dev and prod, so threads load the .ts entry with explicit ts-node registration — validated in all three runtimes (nodemon dev replica end-to-end, prod image inputs, jest real-thread suites).
• Retry-safe streams: the fan-in writer retries inserts with the same rows + same dedup token; the stream is constructed fresh inside insertJsonRows per attempt, so retries never see a consumed stream.
• Helm: telemetryDecodeThreads exposed on both app and worker deployments (app pods consume the telemetry queue on the default worker.enabled=false topology); env renders only when explicitly set so charts defer to the adaptive default. Also fixes a latent values.schema.json gap (additionalProperties: false would have rejected anyone setting the knob).

Safety nets for default-on

Threads spawn lazily on the first ≥8 KiB OTel payload (idle/non-telemetry pods pay nothing); 1-effective-CPU pods resolve to 0 threads; a broken pool trips the breaker and degrades to the pre-pool inline path; all pool rejections are BullMQ-retryable; TELEMETRY_DECODE_THREADS=0 and CLICKHOUSE_STREAMED_INSERTS=false are no-deploy kill switches. When raising thread counts, raise the pod CPU request/limit to match; each thread is a V8+ts-node isolate — budget RSS accordingly.

Testing

102 tests across 9 suites (8 new), real worker threads where it matters, --detectOpenHandles clean:

• inline-vs-thread equivalence over the product × format × encoding matrix (real proto encodings); nonzero-byteOffset slab-view decode
• pool mechanics: ≤N in flight, id correlation, terminate-mid-decode → reject + respawn + recover, breaker trip and heal-after-cooldown, no-hung-promise shutdown with queued work
• streamed inserts: byte-identity against the vendored encoder, per-attempt stream freshness, settings/dedup-token passthrough pinned in both modes, error propagation, plus a guarded live integration test (verified against a real ClickHouse 26.7)
• CpuCount cgroup parsing (v2/v1/fallback/garbage/never-throws) and the adaptive clamp table; routing and config contracts

No new compile errors vs the master baseline. Both change sets adversarially reviewed by multi-lens agent passes with per-finding refuters; final review: zero confirmed findings (earlier rounds' findings — Helm app-pod exposure, schema gap, three test gaps — are all fixed in this PR).

🤖 Generated with Claude Code

https://claude.ai/code/session_01L1ioPKzqvWG9umQPpCryE2

[simlarsen]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.