[RFC]: A Strategic Framework for Extensibility and Innovation in vLLM

[yarongmu-google]

Motivation.

TL;DR:

This proposal outlines a strategic shift in how vLLM manages components and extensions. vLLM's support for hardware- and model-specific plugins has been a cornerstone of its growth, empowering the community to greatly expand the project's hardware coverage and functionality. To build upon this foundation and extend that same flexibility across the entire application, we propose evolving from this specific plugin model to a comprehensive Dependency Injection (DI) framework.

This transition will solve the underlying challenges of brittle, out-of-tree extensions and, more importantly, provide a robust architectural foundation. Adopting DI will accelerate iterative development, simplify testing, and pave the way for advanced capabilities like A/B testing. We recommend a pragmatic, phased adoption, starting with a lightweight simulation of DI to demonstrate value before integrating a full-fledged framework like dependency-injector.

The Common Goal: Stable and Decoupled Components

At their core, both the existing plugin ideology and a formal DI framework share the same fundamental goals:

• Stable Component APIs: To create clearly defined, versioned interfaces that allow the vLLM core and its extensions to communicate reliably.
• Decoupled Architecture: To allow components to be developed, tested, and modified in isolation without causing cascading failures.
• Enable Out-of-Tree Innovation: To empower the community and hardware vendors to build and maintain custom extensions without constant friction from upstream changes.

The current plugin system is a targeted solution. A DI framework is the generalization of this principle, applying it across the entire application to build a truly modular system.

Beyond Plugins: The Strategic Benefits of a DI Framework

A DI framework offers capabilities that a simple plugin/registry system cannot, providing strategic advantages for a project of vLLM's complexity.

1. Accelerates Iterative Development and Experimentation A key benefit of DI is the ability to rapidly swap component implementations. This is a massive accelerator for research and development.

   • Example: A researcher could develop a new experimental LookaheadScheduler and, with a single line change in a DI container, have the entire vLLM engine run with it, directly comparing its performance against the existing VLLMScheduler. This avoids complex refactoring and allows for fast, iterative testing of new ideas.

2. Paves the Way for Advanced Capabilities A decoupled architecture managed by a DI container is a prerequisite for more advanced operational features.

   • A/B Testing: With DI, it becomes feasible to deploy two versions of a component simultaneously. For instance, we could route 5% of requests to a new TrieSampler while 95% use the standard ArgMaxSampler to compare performance, correctness, and resource usage in a live environment.
   • Dynamic Configuration: Easily reconfigure the entire application stack for different use cases (e.g., a low-latency setup vs. a high-throughput setup) by simply loading a different DI container profile.

3. Holistic Dependency Management and Explicit Lifecycles DI gracefully manages the entire object graph and provides critical control over resource lifetimes, which is essential for performance and stability in vLLM. (This includes the previously discussed benefits of managing complex dependencies and object scopes like Singletons).

4. Radically Simplified Testing DI's ability to inject mock objects simplifies unit and integration testing, allowing for robust verification of components in isolation without requiring live hardware.

Recommended Python DI Frameworks

Python's ecosystem offers several excellent, mature options:

• dependency-injector: The most feature-rich and popular framework. Its explicit, container-based approach is highly analogous to enterprise-grade frameworks in other languages and is well-suited for managing the complexity of a project like vLLM.

• punq: A newer, lightweight, and simple alternative that leverages modern Python type hints for a clean and "Pythonic" developer experience.

• FastAPI's Depends: While tied to the FastAPI web framework, its model of declaring dependencies in function signatures is highly ergonomic and serves as a great example of modern DI in Python. It would be the natural choice for any future vLLM API server.

Proposed Change.

A Phased Adoption Approach

To de-risk the transition and demonstrate value incrementally, we propose the following phased approach:

Phase 1: Simulate DI with Module Imports (The Service Locator Pattern) This initial phase requires no new libraries and minimizes disruption.

1. Create a central service module (e.g., vllm.core.services).

2. Define the default implementations in this module. For example: from vllm.core.scheduler_v2 import VLLMScheduler as Scheduler.

3. Refactor the codebase to import components from this central module (from vllm.core.services import Scheduler) instead of their concrete locations.

4. To swap an implementation, a developer only needs to change the import statement in this single file.

This phase allows the team to experience the benefits of centralized dependency management and easy swapping, gauging the value of DI with minimal commitment.

Phase 2: Introduce a DI Framework for New Development Once the benefits are clear, we would:

1. Select and add a formal DI framework (e.g., dependency-injector) as an official project dependency.

2. Mandate its use for all new complex components and modules, establishing it as the architectural standard moving forward.

Phase 3: Gradual Refactoring of Existing Code This is a long-term, opportunistic phase. Critical existing components (Engine, Worker, etc.) can be migrated to the DI framework over time during regular maintenance or feature development, progressively reducing technical debt without halting progress.

This approach balances immediate needs with a strategic vision, ensuring vLLM's architecture remains as innovative as the models it serves.

Feedback Period.

24/7

CC List.

@WoosukKwon @mgoin

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[xuechendi]

Great idea, we also realized that current plugin design somehow constrained the flexibility to make a HW-specific plugin. Including:

1. plugin custom scheduler
2. plugin custom quantization method

My question:

1. What will be the scope for this Proposal, which modules planned to be included?
   Scheduler, sampler, quantization/unquantization layers?

2. what will be the impact to custom_op.py?
   Are you going to add a new 'central service module' for custom_op? Will that change the way for current plugin to override custom layers?
   This is our proposed changes to custom_op for providing custom layer impl from plugin, [custom_op][vllm-plugin] update custom_op class to use op_registry #19164 , if we follow this proposal, what will be different to custom_op.py?

3. What will be the estimated time frame for this proposal?
   Our goal is to bring necessary update listed in [RFC]: Enhancing vLLM Plugin Architecture #19161 to vllm as soon as possible, so we can complete our timeline for migrating HPU into vllm plugin by v0.10. Wanted to know if we are able to leverage changes in this proposal as well?

[houseroad]

Thanks for the proposal. Improving vLLM's extensibility is super important.

Wondering if you would like to give some example about how to leverage dependency-injector or similar library to support customized component?

[yeqcharlotte]

Glad to see more proposals about improving vLLM's extensibility!

Are scheduler and sampler the main components you are looking to extend? While I agree Scheduler needs better interface to support more policies and spec decoding ideas and there should probably be orthogonal modularization effort to make concrete implementation easier to be replaced.

Could you share some examples why vllm.core.services is necessary on top of existing EngineCore?

[yarongmu-google]

Thanks @xuechendi for the insightful questions :) Please see my response inline :)

1. What will be the scope for this Proposal, which modules planned to be included?
   Scheduler, sampler, quantization/unquantization layers?

DI is a generic framework to introduce modularity, so it can applied in all components, including scheduler, sampler, quantization layers - or if someone somehow needs a new frontend, DI can achieve that too. My proposal is to align on a framework, and leaving what components are in scope precisely to the individual component developers :)

2. what will be the impact to custom_op.py?
   Are you going to add a new 'central service module' for custom_op? Will that change the way for current plugin to override custom layers?
   This is our proposed changes to custom_op for providing custom layer impl from plugin, [custom_op][vllm-plugin] update custom_op class to use op_registry #19164 , if we follow this proposal, what will be different to custom_op.py?

DI and custom_op.py are not competing ideas; they are complementary.

The op_registry in your pull request is a fantastic example of the Service Locator pattern, which is in fact the phase 1 of my proposal (ie a cheap way to experiment DI to gauge its benefits). My proposal advocates for generalizing this approach in Phase 2 w/ a real DI framework that manages dependencies in stead of relying on individual service locator patterns coded in the individual components.

Assuming we move forward with DI, in Phase 1, we wouldn't change how custom_op.py or your op_registry works - you are already there :) In Phase 2, (i.e. move to a full DI framework), we could enhance the op_registry: instead of just registering classes, the registry could be managed by the DI container. This would allow for more complex dependency management for custom operations, such as injecting different logging or configuration services into your custom layers.

3. What will be the estimated time frame for this proposal?

Given the flexibility of DI, I think it can be achieved gradually as development goes. Assuming we reach community consensus, then changes like yours (#2 above) is already moving toward DI, so you already started :) At some point I am happy to introduce a central container on top of your change - that can be done any time following migration best practices; though the concrete timeline of moving to Phase 2 depends on consensus timeline.

Our goal is to bring necessary update listed in [RFC]: Enhancing vLLM Plugin Architecture #19161 to vllm as soon as possible, so we can complete our timeline for migrating HPU into vllm plugin by v0.10. Wanted to know if we are able to leverage changes in this proposal as well?

Yes, you totally can! Setting up DI (ie directly jump to Phase 2) is lightweight; but even if you proceed as you plan now, that's the same as Phase 1 of my proposal so no wasted work or anything :)

Please let me know if the above makes sense.

[yarongmu-google]

Hi @houseroad,

Thanks for your question. Please see my response inline :)

Wondering if you would like to give some example about how to leverage dependency-injector or similar library to support customized component?

Here is the pseudo code (I didn't run it) for an example where one implementation injects a worker, a scheduler and a sampler at the same time into vLLM. Obviously you can inject 1, 2 or 100 components in one impl; I just made up 3 for the purpose to demonstration. The better thing, is that in fact a developer can even choose someone else's components to inject - so for example, in one impl, you can run the GPU's scheduler, the TPU's worker, and the HPU's sampler. (I don't think anyone will do that - again this is only for demo purposes).

Step 1. vLLM needs a container that manages all the injectable components - this needs to be done only once for the whole project

from dependency_injector import containers, providers
from vllm.worker.cuda_worker import CudaWorker
from vllm_plugins.tpu.tpu_worker import TpuWorker
from vllm.engine.llm_engine import LLMEngine  # Import the engine

class Container(containers.DeclarativeContainer):
    config = providers.Configuration()

    # --- Worker provider ---
    worker = providers.Factory(...)
    
    # --- Scheduler provider ---
    scheduler = providers.Factory(...)

    # --- Sampler provider ---
    sampler = providers.Factory(...)

    # --- Engine provider ---
    # The container now also knows how to create the LLMEngine.
    # It will automatically resolve and inject the worker, scheduler, etc.
    engine = providers.Factory(
        LLMEngine,
        worker=worker,
        scheduler=scheduler,
        sampler=sampler,
    )

Step 2. Create the components and delegate their management to the container. Let's assume you have CudaWorker and TpuWorker implemented; now update the above code to be:

    worker = providers.Factory(
        lambda worker_type, ...: {
            "cuda": CudaWorker,
            "tpu": TpuWorker,
        }.get(worker_type)(...),
        worker_type=config.worker_type,
        ...
    )

Step 3. Now LLMEngine can be updated to - this only needs to be done once for the whole project

from dependency_injector.wiring import inject, Provide
from vllm.core.containers import Container

@inject
class LLMEngine:
    # The constructor is clean. It only asks for the components it directly uses.
    # It doesn't need to know how to build them.
    def __init__(
        self,
        worker = Provide[Container.worker],
        scheduler = Provide[Container.scheduler],
        sampler = Provide[Container.sampler],
    ):
        self.worker = worker
        self.scheduler = scheduler
        self.sampler = sampler
        # ...

Step 4. Also only needs to be done once for the whole project:

from dependency_injector.wiring import inject, Provide
from vllm.core.containers import Container
from vllm.engine.llm_engine import LLMEngine

@inject
def run_vllm_server(
    # This is the clean pattern you want.
    # This function only asks for the high-level engine.
    # The container handles creating the engine and all of its dependencies.
    engine: LLMEngine = Provide[Container.engine]
):
    print("vLLM server logic started with injected engine.")
    ...

Step 5. Again only needs to be done once for the whole project:

if __name__ == "__main__":
    # 1. Create the container
    container = Container()

    # 2. Read configuration from CLI args, environment variables, or files
    # This is where the decision to use a "tpu" worker is made.
    worker_type_from_config = "tpu" 
    container.config.worker_type.from_value(worker_type_from_config)
    # You could configure everything else here too
    # container.config.scheduler.from_value(...)

    # 3. Wire the container to the modules that need injection
    container.wire(modules=[__name__, "vllm.engine.llm_engine"])

    # 4. Call the clean application logic function.
    # We don't pass any arguments! The @inject decorator and Provide
    # handle resolving the LLMEngine from the container.
    run_vllm_server()
    ...

Basically, post DI developers will focus on developing components for their business logic and delegate component management to DI (ie Step 2); all other changes (Steps 1, 3, 4, 5) will be done once (during migration in Phase 2 of my proposal) for everyone to take advantage of. Obviously as time goes by people may decide more components can be delegated to DI - in that case the repeat Step 1, and then everyone else can just create their counterpart components following Step 2. Steps 3, 4, 5 will not need to be changed.

Please let me know if this makes sense :)

[yarongmu-google]

Hi @yeqcharlotte,

Thanks for your question. Please see my response inline :)

Are scheduler and sampler the main components you are looking to extend? While I agree Scheduler needs better interface to support more policies and spec decoding ideas and there should probably be orthogonal modularization effort to make concrete implementation easier to be replaced.

Oh schedulers and samplers are simply examples I put forward to illustrate my proposal. DI is a framework so can help manage counterpart implementations of pretty much everything. For example, maybe I need a customized sampler, you need a customized engine, etc etc. The point is, we both can seamlessly integrate what our HW/use case need, and we can also choose others' components for reuse, w/o having to duplicately implement the same logic. This is one critical diff between DI and plug-ins - DI makes reusing much easier. For example, if I have a customized sampler_1, and you have a customized engine_1, and you want to use my sampler_1, together w/ vLLM's default kv_cache_manager, then you can run your instance as (engine_1, kv_cache_manager, sampler_1) while I run my instance as (sampler_1).

Could you share some examples why vllm.core.services is necessary on top of existing EngineCore?

Oh vllm.core.services is my made up container for people to register their implementations to be injected or reused. It manages all components available, contributed by the whole OSS community. A more compressive example would be the Step 1 in my answer to the previous question above (though in that answer, it should be called vllm.core.container - please ignore my example naming inconsistencies.) So vllm.core.services is the container that provides an impl when needed, while EngineCore is an impl, which may depend on other impls included in vllm.core.services, and itself can be managed by vllm.core.services too.

Please let me know if this makes sense:)

[maxdebayser]

More extensibility in vLLM would be great. I've been involved in a few proposals to make specific aspects of vLLM pluggable such as #12363 and #12249 and it would be greate to have a consistent way to do this kind of thing across vLLM. That said, I see 2 challenges for a general DI framework within vLLM:

• For DI to work, we need stable interface definitions. Given the fast pace of innovation, this could prove difficult.
• The wiring and configuration process of the components could be tricky and require several passes. For example, there are several settings that the user can set such as prefix caching. The ability to do prefix caching depend both on the model implementation (it has to be a decoder model) as well as the platform capability. Currently there is a lot of logic in config.py to handle all these different corner cases. A good example of this complexity is the _get_and_verify_max_len function.

[yarongmu-google]

Thanks @maxdebayser for your comments. Please see my response inline :)

• For DI to work, we need stable interface definitions. Given the fast pace of innovation, this could prove difficult.

This is in fact exactly why we propose DI, vs extending vLLM's existing Plug-in system, despite that they both implement the DIP principle :) One key differences between DI and Plug-ins is that, while Plug-ins rely on standard, stable interfaces definitions between components, DI doesn't. It supports heterogeneous interfaces between components, eliminating the need for standardized, pre-agreed APIs. Whether we want to go this far is subject to discussion; but from teh perspectives of stable APIs, it's no worse than the existing plug-in mehcnaisms.

• The wiring and configuration process of the components could be tricky and require several passes. For example, there are several settings that the user can set such as prefix caching. The ability to do prefix caching depend both on the model implementation (it has to be a decoder model) as well as the platform capability. Currently there is a lot of logic in config.py to handle all these different corner cases. A good example of this complexity is the _get_and_verify_max_len function.

This is also elegantly handled by DI - the whole point is developers no longer have to manually wire or configure anything. In a mature DI codebase, one may even only see"definitions" w/o any wiring. One can do mix and match as long as the interfaces of involved APIs match (vs a stable API across all implementations, see above). Again, we have the option of not going that far; but the possibility is there.

Please let me know if the above makes sense :)