[Roadmap]: Dynamo roadmap until 0.4.0 (8/6) #762
Description
UPDATE: On 8/12/25, we launched Dynamo v0.4.0, which is our first stable release. In the H2, we plan to GA Dynamo and focus on 5 key areas:
- Performance
- Fault tolerance
- K8 deployment
- Memory management and transfer
- Scheduling with router and planner
We will create a more detailed H2 roadmap and post it as a pinned issue by week of 8/18. As we make progress towards a GA product, we would like to prioritize features based on feedback and share our progress more accurately. Please be on the look out for the H2 roadmap, and thank you so much for your ongoing support.
We're sharing our roadmap leading up to the v0.4.0 release to foster open-source development of Dynamo.
By then, our goal is to make Dynamo production-ready for GenAI inference.
This roadmap is subject to change, and community contributions are highly encouraged. If you're interested in contributing to specific features, please comment on this issue.
To accelerate progress, Dynamo releases will follow a biweekly cadence. Expect a major release monthly (incrementing by 0.1) with minor releases in between.
** UPDATE**: We are pushing back v0.4.0 by two weeks to prioritize performance, fault tolerance, observability, and metrics. We will be descoping application features such as agents and multi-LoRA. Descoped features are crossed out in the following objectives, and new added features are indicated in bold
Key objectives from v0.2.0 until v0.4.0
- LLM support
- Performant disaggregated serving including DeepSeek R1 -> WideEP performance is in progress with inference engines. SGLang & TRT-LLM wideEP scripts are available on Github.
-
Multi-LoRA-> targeted for 0.5 release - Speculative decoding
- Full feature support with TRT-LLM, vLLM, and SGLang
- Multimodal
- Text - image - video model support
- E/P/D disaggregation
-
Multimodal cache
- KV cache manager -> targeting 0.4.1 to support HBM, host memory, and local disk
- KV offloading to multiple levels of memory hierarchy
- Local & Network storage support with most known storage vendors
- Performant multi-turn conversations
- [x ] Planner
- Dynamic allocation of prefill and decode
- SLA requirement based real time performance tuning
**- [ ] Fault tolerance - Request cancellation and migration -> Request cancellation targeted for 0.4.1 release
- Model recovery with faster weights
- [x ] Monitoring for HW and instance load/recovery time**
- Observability and metrics
- OpenTelemetry -> supported through structured logging aligned with OTel
- Metrics for dcgm, frontend, and worker
-
Agents-
Constrained decoding and function calling -
Performant KV offloading and pre-fetching based on predicted time of agent execution -
MCP support
-
- Performance benchmarking
- GPU level metrics
- Energy metrics for TCO calculation
- Validated K8 workflow for deployment
- Scale up to 64 GPUs
- Helm charts and custom operators
- AWS/Azure/GKE support and tutorials -> GKE will be added to 0.4.1 release
Expected timeline
Here are the major features you can expect in our next immediate release. We will provide more details for subsequent releases iteratively to ensure transparency. Please stay tuned for further updates.
-
v 0.2.0 (End of April)
- KV Manager
- Offloading enabled for GPU, host memory, SSD, and network storage
- Planner
- Dynamically allocate prefill and decode
- Validated K8 workflow
- Helm charts and custom operators
- NIXL AWS EFA support and NIXL microbench
- KV Manager
-
v 0.2.1 (Mid May)
- vLLM v1 support and reduced performance overhead
- SGLang integration
- KV Manager
- Offloading enabled for GPU, host memory, SSD, and network storage
- Planner with K8 support
- Multimodal model support
- Functional E/P/D disaggregation with text + image model (Llava 1.5)
- NIXL Mooncake plugin integration
-
v 0.3.0 (Targeted for 6/4)
- Performant Deepseek R1 disaggregated serving with SGLang, TRT-LLM, and vLLM
- SGLang focused on Hopper performance
- TRT-LLM focused on Blackwell performance
- Fault tolerance for Dynamo components
- KV Manager integrated with Dynamo runtime and vLLM
- K8 support
- Model caching across pipelines
- Initial Gitops implementation for rolling updates and zero-downtime deployment
- Multimodal model support
- Performant E/P/D disaggregation with text + image model
- Functional E/P/D disaggregation with text + video model
- Planner
- Provide guide + sweep script to allow user to pick up a good starting configuration based on SLA
- NIXL
- Generic object storage support
- GPU initiated communication
- UCX Resiliency
- Performant Deepseek R1 disaggregated serving with SGLang, TRT-LLM, and vLLM
-
v 0.3.1 (7/1)
- Functional DeepSeek R1 disaggregated serving with wide EP using SGLang
- Functional EPD disaggregation with video model (Llava video 7B)
- Proof of concept inference gateway support
- Prebuilt Dynamo + vLLM container
- Amazon Linux support (x86)
-
v 0.3.2 (7/18)
-
PerformantFunctional R1 disaggregated serving with wide EP using SGLang - Functional R1 disaggregated serving with wide EP using TRT-LLM
- Improved UX experience for simple and direct K8 deployment without using SDK with CLIs.
- Native DCGM and Prometheus integration enables hardware metrics collection and export
- E2E functional KV Block Manager with vLLM capable of offloading to HBM, system memory, and local disk - core code done still working on vLLM integration
- Pre-built containers for vLLM, SGLang, and TRT-LLM
- SLA (TTFT & ITL) based planner
-
-
v 0.4.0 (8/12)
- Native vLLM v1 support
- Improved performance of R1 disaggregated serving with wideEP for SGLang and TRT-LLM
- Enhanced helm chart flexibility and K8 examples for SGLang, TRT-LLM, vLLM
- Request migration
- Structured logging, hierarchical Prometheus metrics registry, and custom metric support
- Multi-modal example with Llama Maverick