[Roadmap] Inference Endpoints Phase 2 Roadmap

[arekay-nv]

We will document the issues that are required to reach key milestones in this document.

How to join us

• Subscribe to our mailing list: link
• Join our Slack: link

Goal

Post-GTC / Q2 2026: Stabilize the benchmarking platform, expand accuracy coverage, harden reliability, and close all known P0 gaps following the GTC demo.

See Phase 1 roadmap: #83

---

ShowStopper

• Accuracy evaluation for LLMs (GPQA, AIME, HLE, etc.) Feat: Add accuracy evaluation for LLMs (GPQA, AIME, HLE etc.) #4
• Throughput/latency parity with MLPerf LoadGen and vLLM benchmark_serving Perf: analyze and reach parity with MLPerf LoadGen, vLLM benchmark_serving (and backend metric server) #8
• Pareto submission clarification (models, datasets, points) Pareto clarification #84

---

Functionality (P0)

• Submission ruleset interaction with config (Phase 4 integration) Feat: define and implement the submission ruleset interaction with config #5
• Define and implement the final report structure Feat: define and implement the final report structure #6
• Analyze roofline of inference endpoints (offline/online/SSE) Perf: Analayze the roofline of the inference endpoints #9
• Pub/Sub implementation for event recorder feat: Pub/Sub implementation for event recorder #183
• Remove unsafe serialization support (security: arbitrary code execution risk) feat: Remove pickle support #184
• EVAL and SUBMISSION test types raise bare NotImplementedError — add user-friendly errors bug: EVAL and SUBMISSION test types raise bare NotImplementedError with no user-friendly message #218

---

Bug Fixes

• PostgreSQL DuplicatePreparedStatement error in recorder Resolve Endpoints PostGres dup element issue #213
• max_throughput mode causes connection timeouts "max_throughput" mode causes connection timeouts #202
• Performance discrepancy between submission config and performance-only config Performance discrepancy between submission config and performance-only config #199
• max_duration_ms ends online test prematurely before all samples are issued [Runtime setting] max_duration_ms: Ends online test prematurely (soon as duration is hit). #197
• target_qps hardcoded to 10.0 in Offline mode instead of None bug: target_qps hardcoded to 10.0 in Offline mode instead of None #219
• RuntimeSettings stores Random objects — benchmarks non-reproducible and non-serializable bug: RuntimeSettings stores Random objects making benchmarks non-reproducible and non-serializable #221
• Errors seen running endpoint tests in local mode errors seen running endpoint tests in local mode #99
• llama.cpp incompatibility (requires OpenAI-compatible server) Endpoints code requires an OpenAI API compatible server (llama.cpp incompatability) #133

---

Accuracy & Datasets (P1)

• MMLU / MMLU-Pro — dataset + scorer + extractor MMLU / MMLU-Pro - dataset + scorer + extractor #176
• MATH500 — dataset integration MATH500 - Dataset integration #177
• Healthbench integration Healthbench integration #178
• Humanity's Last Exam (HLE) integration Humanity's Last Exam - integration #179

---

Performance & Benchmarking (P1)

• System bottleneck tests (client stress tests) Perf: system bottleneck tests #10
• Explain text vs token performance regression on TRTLLM Perf: Explain text vs token perf on TRTLLM #182
• Implement BURST and STEP load patterns feat: Implement BURST and STEP load patterns #217

---

Testing & CI (P0/P1)

• Add GitLab CI to stress test upper bound of endpoint measurement Add GitLab CI to stress test upper bound of the endpoint measurement #138
• SGLang adapter integration tests are permanently skipped test: SGLang adapter integration tests are permanently skipped #220
• KVStore, ServiceLauncher, and MetricsAggregator services lack integration tests test: KVStore, ServiceLauncher, and MetricsAggregator services lack integration tests #222

---

UI/UX & Visualization (P1)

• Runtime visualization of the benchmark Feat: Implement the runtime visualization of the benchmark #7