Skip to content

perf(gateway): reduce per-turn SQLite span amplification #1685

Open

Description

@BYK

Problem

Ordinary turns create roughly 600–1,060 Sentry SQLite child spans. Individual queries are fast, but constructing, retaining, scrubbing, and exporting this many in-process spans can amplify allocation, GC, and response overhead during already expensive turns.

Evidence

  • 455,061 database spans observed; p50 0.049 ms and p95 0.621 ms.
  • Representative ordinary traces contain 619–1,062 SQLite children.
  • Slow HTTP and model spans coexist with fast SQL, so database latency does not explain the turn duration, but instrumentation volume remains material.

Required investigation

Measure allocation and latency with current spans, aggregated metrics, and sampled spans under equivalent deterministic load. Keep visibility into real failures and read-path latency while reducing per-query fan-out if evidence confirms material overhead.

Acceptance criteria

  • Benchmark quantifies CPU, heap, and payload cost of current instrumentation.
  • Any replacement keeps fixed-message errors and reviewed metrics.
  • No raw SQL parameters, paths, content, or arbitrary errors enter telemetry.
  • Telemetry never throws or blocks the read path or idle scheduler.

Concrete implementation sequence

Related incident: #1708. Reuse #1733 when available; this investigation can also start with the current deterministic fixtures.

  1. Run the same seeded long-session workload with the current SQLite child spans, a local no-export instrumentation control, and an aggregated/sampled prototype. Keep database contents, worker counts and provider stubs fixed.
  2. Measure CPU, allocation/retained heap, span count/export bytes and post-decode/health latency over repeated samples. Separate exporter/network cost from span construction/scrubbing.
  3. If the comparison identifies material overhead, implement the smallest aggregation/sampling change and preserve real failure visibility plus read-family latency.
  4. Commit the command and numeric report, with regressions for bounded retained instrumentation and exporter failure.

#1732 adds bounded phase/worker attribution; it does not replace or duplicate this span-volume investigation. No OTLP migration is required (#1614).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions