[Roadmap] Inference Endpoints 2026-Q1 Roadmap

[arekay-nv]

We will document the issues that are required to reach key milestone in this document.

How to join us

• Subscribe to our mailing list: link
• Join our Slack: link

Goal

GTC/V6.0 release software and initial benchmarking results by 4/2/2026.

Functionality

• Accuracy evaluation support Feat: Add accuracy evaluation for LLMs (GPQA, AIME, HLE etc.) #4
• Config refactor with submission ruleset support (e.g. MLPerf Inference v6.0 ruleset) Feat: define and implement the submission ruleset interaction with config #5
• MLPerf submission checker compatibility Make results compatible to MLPerf submission checker for 6.0 #29

Model support

• llama2-70B llama2 -70b support and calibration #49
• llama3-8b llama3.1-8b support and calibration #51
• llama3-405B LlaMa3-405b support and calibration #50
• gpt-oss #112
• DeepSeek #113

Performance Tuning

• Throughput/latency measurement Parity with MLPerf LoadGen, vLLM benchmark_serving Perf: analyze and reach parity with MLPerf LoadGen, vLLM benchmark_serving (and backend metric server) #8
• Maximum concurrency and SSE per second testing and measurement Perf: Analayze the roofline of the inference endpoints #9
• System bottleneck testing (client stress tests) Perf: system bottleneck tests #10

UI/UX, input/output

• Structured throughput, latency and accuracy report Feat: define and implement the final report structure #6
• Runtime visualization Feat: Implement the runtime visualization of the benchmark #7

Documentation

• Full documentation of the inference-endpoints usage.
• CLI documentation Documentation: Hierarchy of CLI arguments #37

[nvzhihanj]

Closing this Q1 2026 Roadmap issue as it has been superseded by the Phase 2 Roadmap tracked in #223. Please follow #223 for current roadmap and planning updates.