Repository navigation
Post-thread-per-core performance program: cross-gateway benchmark comparison and a prioritized hot-path optimization list #905
Description
Closed: this line of work is being tracked in internal planning. Keeping a de-branded summary here for reference.
Background
Public AI-gateway benchmark leaderboards run entrants on identical dedicated hardware (m7g.4xlarge Graviton3: 4 cores pinned for the gateway, the rest split between load generation and a local mock upstream) in the default shipped configuration — no tuning.
#891 (thread-per-core workers) roughly doubles our 4-core saturation throughput — 24.0K → 45.2K rps (c=128, 4 pinned x86 vCPUs, local mock, +88%) — by eliminating the two cross-thread handoffs per request. Published figures for the fastest entrants still sit meaningfully above that. The numbers are not apples-to-apples (different hardware, mocks, concurrency), so the first step of any follow-up is a same-rig measurement.
Goal. Not to win a synthetic forwarding race — a bare forwarder always wins that, and we ship guardrails, budgets, rate limiting, caching, telemetry, and protocol translation. The goal is to identify which techniques used by faster gateways are compatible with our feature set, and land them incrementally, each with before/after numbers.
Directions under evaluation
Baseline: post-#891 HEAD, thread-per-core on.
- Same-rig measurement using the public-leaderboard methodology (extends the Graviton (m7g.4xlarge) validation of thread-per-core serving before release #902 Graviton gate), with a flamegraph per entrant (Establish the on-CPU profiling workflow: adopt cargo-flamegraph and add a dedicated profiling build profile #847 workflow).
- Default-config request-path cost audit: per-request µs cost for everything on the hot path in the default config; keep / make lazy / propose default-off per item.
- Known user-space hot spots from feat(proxy): serve from thread-per-core workers with per-worker upstream pools #891's profile: per-request
getrandomfor request-id and the metrics-macro registration path (~11% of CPU combined). - Release-profile review (LTO / codegen-units / strip).
- Dependency and feature audit (
default-featuressweep, duplicate HTTP/TLS stacks). - Same-protocol byte passthrough where ingress and egress protocols match.
- Contention audit of remaining cross-worker shared state (Exporter delivery task pins itself to one worker in thread-per-core mode #901).
Constraints
- The feature set is not up for negotiation; "delete the feature" is not an optimization. Making a feature's idle cost zero when unconfigured is.
- One PR per item, each with same-rig before/after numbers.
- Anything that changes a shipped default needs product sign-off plus a docs story before it lands.
Activity
coderabbitai commentedon Aug 7, 2026
🔗 Related PRs
#816 - feat(retries): move the retry budget onto the model and apply it everywhere [merged]
#835 - feat(headers): request-context variables in default_headers + client-header forwarding allowlist [merged]
#853 - feat(config): request_body_limit_bytes defaults to 0 = no cap; uniform 413 surface [merged]
#870 - feat(obs): split the TTFT bucket set from e2e latency and make both configurable [merged]
#894 - docs: align AISIX product terminology [merged]
📝 Issue Planner
Check the box below or use the
@coderabbitai plancommand to generate an implementation plan and prompts that you can use with your favorite coding assistant.🧪 Issue enrichment is currently in open beta.
You can configure auto-planning by selecting labels in the issue_enrichment configuration.
To disable automatic issue enrichment, add the following to your
.coderabbit.yaml:💬 Have feedback or questions? Drop into our discord!
[-]Post-thread-per-core performance program: same-rig comparison with Busbar and a prioritized hot-path optimization list[/-][+]Post-thread-per-core performance program: cross-gateway benchmark comparison and a prioritized hot-path optimization list[/+]membphis commentedon Aug 7, 2026
Tracking this work in internal planning going forward.