[Roadmap] Inference Endpoints 2025Q4 GA Roadmap

[nvzhihanj]

We will document the issues that are required to reach the GA milestone in this document.

How to join us

• Subscribe to our mailing list: link
• Join our Slack: link (TBD)

Goal

GA release software and initial benchmarking results by 12/19/2025.

Functionality

• Accuracy evaluation support Feat: Add accuracy evaluation for LLMs (GPQA, AIME, HLE etc.) #4
• Config refactor with submission ruleset support (e.g. MLPerf Inference v6.0 ruleset) Feat: define and implement the submission ruleset interaction with config #5
• Record first-chunk for accurate TTFT and TPOT recording. Feat: Add first-chunk support for the http client and recorder #15
• Error logging and handling of the queries Feat: error logging and handling of the responses #16

Performance Tuning

• Throughput/latency measurement Parity with MLPerf LoadGen, vLLM benchmark_serving Perf: analyze and reach parity with MLPerf LoadGen, vLLM benchmark_serving (and backend metric server) #8
• Maximum concurrency and SSE per second testing and measurement Perf: Analayze the roofline of the inference endpoints #9
• System bottleneck testing (client stress tests) Perf: system bottleneck tests #10

UI/UX, input/output

• Structured throughput, latency and accuracy report Feat: define and implement the final report structure #6
• Runtime visualization Feat: Implement the runtime visualization of the benchmark #7

Documentation

• Full documentation of the inference-endpoints usage.
• Add documentation of how to ramp up server LLM endpoints (e.g. vLLM, SGLang) for testing Documentation: Add example of how to ramp up vLLM/SGLang endpoints for benchmarking #11