Skip to content

test(perf): add a reproducible long-session workload with background pressure #1733

Description

@BYK

Parent incident: #1708

Priority: First: baseline and shared verification

Problem

The #1710 benchmark demonstrates semantic/window preparation improvements but excludes LTM/embedding retrieval and upstream inference, and its database reopen occurs in the same process. It cannot establish responsiveness for the reported 7,228-message workload while maintenance runs.

Implementation plan

  1. Extend the existing benchmark fixtures with seeded tool-heavy Responses sessions around 5,580 and 7,228 messages and a roughly 180k–200k-token active window. Include large tool outputs, provenance, current-turn tool pairs, and configurable knowledge/vector/backlog sizes.
  2. Exercise the real gateway and SQLite/read-worker path against a deterministic local upstream stub. Include both append continuation and a genuinely fresh serving-process restart.
  3. Add independent scenarios for idle backlog, foreground arrival during a backlog, unavailable/hung embeddings, unavailable/slow read workers, cancellation, and concurrent sessions. Use provider stubs for deterministic CI and an opt-in native-provider run for diagnosis.
  4. Report decode time separately from decoded-request-to-upstream overhead, health latency during load, process CPU, queue depth/age, retained memory, fallback counts and query execution counts. Reuse perf(observability): attribute memory preparation and vector worker latency #1732 when available without making it a build dependency.
  5. Commit the generator, commands and machine-readable baseline in the repository. Permit a local production-database copy workflow, but never commit/upload user transcripts or a live database.

Acceptance criteria

  • A single command reproduces the baseline without paid APIs or a private dataset.
  • Restart tests start a new process, and workload parameters/seeds/runtime/build identity are recorded.
  • Deterministic assertions cover bounded queues, cancellation, output/provenance/tool-pair parity and durable-write ordering.
  • Timing comparisons use warmup/repeated samples and report hardware; CI does not depend on brittle universal millisecond thresholds.
  • The report tracks the proposed integration goal of p95 post-decode Lore overhead below 2 s and health p95 below 250 ms on a stated reference machine, including degraded-provider scenarios. These are targets, not current measurements.

Dependencies and boundary

Can start independently and reuse the existing #1710 fixtures. #1731 and #1732 improve diagnosis but are not hard prerequisites. This is a performance/responsiveness harness, not the behavioral memory-evaluation suite in #1211 or heap-owner investigation in #1683.

Hard prerequisites

None. This issue can be implemented independently.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions