feat(flowcontrol): Add Foundational Types and Architecture #997

LukeAVanDrie · 2025-06-16T21:09:02Z

Description

This PR introduces the foundational building blocks for the new Flow Control system, designed to manage priority, fairness, and queuing for inference workloads.

The goal is to provide a sophisticated mechanism for managing diverse SLOs and preventing issues like Head-of-Line blocking and system overload, which are not addressed by simple FCFS temportal scheduling. This is a relatively sophisticated and complex system that will need to be split across several PRs. This initial submission lays the groundwork for the entire module by establishing its core concepts and data contracts (basically all types with no cross-package dependencies).

This work tracks issue #674

For historical context, the initial KEP-like proposal can be found here. Please note that the design has evolved significantly, and this PR represents the most current architecture (or at least a small slice of it). I will do some documentation cleanup once more of these PRs with the canonical architectural decisions are out for review.

Contribution

This PR includes two main pieces:

Top-Level README.md: Provides a high-level overview of the Flow Controller, including:
- A clear Motivation for why this component is necessary.
- A high-level Architectural Diagram showing the component interactions and request flow.
- A summary of the core Architectural Pillars (Controller, Framework, Registry, etc.).
The types Package: This new package establishes the core "vocabulary" for the entire Flow Controller module. It includes:
- Request Lifecycle Interfaces (FlowControlRequest, QueueItemAccessor, QueueItemHandle).
- Final Outcome Reporting via the QueueOutcome enum and a structured set of error types (ErrRejected, ErrEvicted).
- The FlowSpecification interface for defining workload identity and priority.
- A README.md detailing the concepts within the types package.

Review Focus

As this is a foundational PR, I'm particularly looking for feedback on:

Does the high-level architecture and motivation make sense?
Are the data models in the types package clear, logical, and sufficient for the tasks ahead?
Is the vocabulary (e.g., QueueOutcome, the error hierarchy) well-defined and intuitive?

Future PRs will build upon these types to implement the framework, registry, and controller packages.

netlify · 2025-06-16T21:09:08Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`25a1e08`
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/6851efe911955d0008622782
😎 Deploy Preview	https://deploy-preview-997--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

k8s-ci-robot · 2025-06-16T21:09:13Z

Hi @LukeAVanDrie. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

LukeAVanDrie · 2025-06-16T21:12:02Z

I have the followup PRs for the rest of the type system that contextualize this queued up, but I am currently on a 🚂 and my connection is spotty. Will send those out soon.

The next PR will detail the framework package (all the extension points) which should be a greater source of discussion compared to this PR. I will then have followups for the ports (saturation detector and flow registry) and the core execution engine (flow controller). These will have much more detailed architecture diagrams as well. The top-level README is intended to have the high level view only.

danehans · 2025-06-16T21:19:07Z

/ok-to-test

Introduces the foundational packages for the new Flow Controller component. This change includes: - The top-level README outlining the motivation, high-level architecture, and component pillars. - The `types` package, which defines the core data contracts, request lifecycle interfaces, error-handling vocabulary, and final outcome enums for the entire module. This foundational PR establishes the core concepts and data models upon which the rest of the Flow Controller implementation will be built.

ahg-g

Do you have a followup PR that uses those types, it is easier to reason about those interfaces and types when they are actually used.

ahg-g · 2025-06-20T17:06:44Z

pkg/epp/flowcontrol/README.md

+The Flow Controller is a sophisticated library designed to solve these problems. It acts as a crucial gatekeeper that
+decides *if* and *when* a request should proceed to be scheduled. Its primary mission is to enable predictable, fair,
+and efficient utilization of shared backend resources by enforcing prioritization, applying fairness policies, managing
+request queuing under saturation, and orchestrating displacement (the eviction of lower-priority queued items to make


I would use the word shedding instead of eviction

I named the specific extension point "displacement". I documented the justification in: https://github.com/LukeAVanDrie/gateway-api-inference-extension/blob/flow-control/pkg/epp/flowcontrol/framework/README.md#terminology-dispatch-vs-displacement.

Eviction, preemption, and shedding were also considered.

ahg-g · 2025-06-20T17:21:09Z

pkg/epp/flowcontrol/README.md

+It is designed for extensibility, allowing custom logic for policies and queuing mechanisms to be plugged into a robust,
+high-performance orchestration engine.
+
+### Role in the Gateway API Inference Extension


I recommend to define goals and non-goals, here is my rough list:

Goals:

Enable priority and fairness across workloads.

Optimize request scheduling by shifting a portion of request queueing from the model server to a centralized queue.

O(seconds) average and O(single digit minutes) tail queueing time

Non-goals:

Persistence: Queueing is handled in the endpoint picker's memory.

Scale: Scale is limited by available memory and number of ext-proc connections of the endpoint picker and L7LB. While they can horizontally scale, maintaining connections for many minutes to hours is not a goal.

A substitute for a message queue: The queue manages requests with open connections via the L7LB and is not intended for asynchronous request handling.

ahg-g · 2025-06-20T17:54:26Z

pkg/epp/flowcontrol/README.md

+    class A2 downstream_err;
+    class D,E,F,P,SD default;
+```
+


We should clearly distinguish between request flow (the top one?) and configuration flow (the bottom one?).

Ack, I will add some controller (e.g., a k8s operator) that configures the Flow Registry.

We don't want to completely detach from epp, so I don't think we want to describe a generic k8s operator that does this.

I am just asking to distinguish between the arrows that are describing a request flow (traversed for each request) and the ones that are configuration (registering plugins and whatnot). I would split this actually into two diagrams if necessary. I would like to see the data flow separately from the configuration flow

I would like to see the data flow separately from the configuration flow

++ they are different concerns & should be able to be reasoned about seperately

ahg-g · 2025-06-20T17:57:04Z

pkg/epp/flowcontrol/README.md

+    concurrent-safe request storage. It uses a `QueueCapability` system that allows for diverse and extensible queue
+    implementations (e.g., FIFO, Priority Heap) while maintaining a stable interface.
+
+4.  **The `FlowRegistry` (`./registry`, `./ports`)**: This is the stateful control plane of the system. It manages the


what is a port in this context?

I am using a "Ports and Adapters" (or "Hexagonal") architecture pattern here. The ports in this flow control system are documented here: https://github.com/LukeAVanDrie/gateway-api-inference-extension/blob/flow-control/pkg/epp/flowcontrol/ports/README.md.

ahg-g · 2025-06-20T18:01:35Z

pkg/epp/flowcontrol/types/README.md

@@ -0,0 +1,33 @@
+# Flow Control Core Types


I recommend having this documentation in the code as comments, I worry that as the code evolves, such detailed documentation will quickly go out of sync.

++.
this PR includes a lot of documentation.
I strongly recommend leaving only the main README and remove all the smaller docs.
two main patterns I recommend to adopt:

as Abdullah mentioned, document these kind of things in the code as comments.

write your code as a self explainable code as much as possible. that means, using meaningful function names and variable names, structuring your code in a way that is easy to follow (and maintain), etc.

such a lot of documentation on multiple markdown files usually gets out of sync very quickly.

ahg-g · 2025-06-20T18:07:37Z

pkg/epp/flowcontrol/types/flow.go

+	// Priority returns the numerical priority level currently associated with this flow within the Flow Registry.
+	//
+	// Convention: Lower numerical values indicate higher priority.
+	Priority() uint


This is great, so we are not limited to discrete values, correct?

Yep; however, the priority bands (defining the possible values of Priority()) are startup-time configured in the Flow Registry. Most of the system supports dynamic updates, just not this.

Flows can migrate between priorities at runtime, you just cannot change what the priority options are.

why do we have this limitation?

ahg-g · 2025-06-20T18:09:31Z

pkg/epp/flowcontrol/types/request.go

+// An object implementing this interface is the primary input to `FlowController.EnqueueAndWait()`. The controller then
+// wraps this object with its own internal structures (which implement `QueueItemAccessor`) to manage the request's
+// lifecycle without modifying the original.
+type FlowControlRequest interface {


Do we expect to have more than one implementation of this interface? if so, is it really necessary to have this as an interface vs a struct?

I left it this way so this can be imported outside EPP. All of this is functional as a portable library. Also, I have an escape hatch with OriginalRequest that allows back casting to get richer request metadata not commonly shared between policies. This is useful for prototyping for advanced use cases. I think there is some value in allowing any type implementing this interface to be enqueued.

This also makes testing a bit easier.

nirrozenbaum · 2025-06-22T20:38:25Z

pkg/epp/flowcontrol/types/errors.go

+// When returned by `FlowController.EnqueueAndWait()`, these specific errors will typically be wrapped by `ErrRejected`.
+var (
+	// ErrNilRequest indicates that a nil `types.FlowControlRequest` was provided.
+	ErrNilRequest = errors.New("FlowControlRequest cannot be nil")


is there any use case where request is nil? isn't this verified in higher layers before FlowControl?
I understand your code makes no assumptions on input validity, but we need to decide when is it ok to remove obvious checks.. if we initialize the request in GIE, then it's probably won't be nil :)

nirrozenbaum · 2025-06-22T20:41:15Z

pkg/epp/flowcontrol/types/flow.go

+// FlowSpecification defines the configuration of a logical flow, encapsulating its identity and registered priority.
+//
+// A FlowSpecification acts as the registration key for a flow within the Flow Registry.
+type FlowSpecification interface {


nit: why not just Flow?

nirrozenbaum · 2025-06-22T20:51:19Z

pkg/epp/flowcontrol/types/request.go

+}
+
+// QueueItemHandle is an opaque handle to an item that has been successfully added to a `framework.SafeQueue`. It acts
+// as a key, allowing the Flow Controller to perform targeted operations (like removal) on a specific item without


"it acts as a key".
this sounds a bit strange. I'd expect a FlowControlRequest to have a RequestID (or just ID) and use that identifier as the key for the request.
I've seen you defined ID in the Request but wrote it's optional and not used internally.

it's pretty hard to comment on code that still not here, but overall using an interface as a key instead of the ID sounds against the initial intuition.

might be good to get back to this point after seeing the implementation PR.

kfswain · 2025-06-23T15:30:49Z

This seems reasonable enough, but I would love to see an implementation PR accompanying the type system. It's hard to reason about the extension points we care about without a working implementation

nirrozenbaum · 2025-06-26T10:47:07Z

This seems reasonable enough, but I would love to see an implementation PR accompanying the type system. It's hard to reason about the extension points we care about without a working implementation

++. as mentioned, it is easier to reason about the interfaces and types when they are actually used.
this can be merged and we can iterate on the definitions in the next PR if needed.

/lgtm
/approve

k8s-ci-robot · 2025-06-26T10:47:15Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: LukeAVanDrie, nirrozenbaum

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [nirrozenbaum]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Introduces the foundational packages for the new Flow Controller component. This change includes: - The top-level README outlining the motivation, high-level architecture, and component pillars. - The `types` package, which defines the core data contracts, request lifecycle interfaces, error-handling vocabulary, and final outcome enums for the entire module. This foundational PR establishes the core concepts and data models upon which the rest of the Flow Controller implementation will be built.

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 16, 2025

k8s-ci-robot requested review from Jeffwan and kfswain June 16, 2025 21:09

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jun 16, 2025

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jun 16, 2025

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 16, 2025

LukeAVanDrie force-pushed the flow-control-system branch from c86d3d8 to 0645012 Compare June 17, 2025 22:11

LukeAVanDrie force-pushed the flow-control-system branch from 2c6952e to 25a1e08 Compare June 17, 2025 22:44

ahg-g reviewed Jun 20, 2025

View reviewed changes

nirrozenbaum reviewed Jun 22, 2025

View reviewed changes

nirrozenbaum added the gie-area/flowcontrol Categorizes an issue or PR as relevant to GIE FlowControl. label Jun 24, 2025

k8s-ci-robot assigned nirrozenbaum Jun 26, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 26, 2025

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 26, 2025

k8s-ci-robot merged commit b4d1e67 into kubernetes-sigs:main Jun 26, 2025
9 checks passed

LukeAVanDrie deleted the flow-control-system branch June 26, 2025 21:36

feat(flowcontrol): Add Foundational Types and Architecture #997

feat(flowcontrol): Add Foundational Types and Architecture #997

Uh oh!

Conversation

LukeAVanDrie commented Jun 16, 2025

Description

Contribution

Review Focus

Uh oh!

netlify bot commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for gateway-api-inference-extension ready!

Uh oh!

k8s-ci-robot commented Jun 16, 2025

Uh oh!

LukeAVanDrie commented Jun 16, 2025

Uh oh!

danehans commented Jun 16, 2025

Uh oh!

ahg-g left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LukeAVanDrie Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LukeAVanDrie Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LukeAVanDrie Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nirrozenbaum Jun 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kfswain commented Jun 23, 2025

Uh oh!

nirrozenbaum commented Jun 26, 2025

Uh oh!

k8s-ci-robot commented Jun 26, 2025

Uh oh!

Uh oh!

Uh oh!

netlify bot commented Jun 16, 2025 •

edited

Loading

LukeAVanDrie Jun 20, 2025 •

edited

Loading

LukeAVanDrie Jun 20, 2025 •

edited

Loading

LukeAVanDrie Jun 20, 2025 •

edited

Loading

nirrozenbaum Jun 22, 2025 •

edited

Loading