feat: Introduce pluggable queue framework

pkg/epp/flowcontrol/framework/doc.go

@@ -0,0 +1,27 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+// Package framework defines the core plugin interfaces for extending the `controller.FlowController`.
+//
+// It establishes the contracts that custom logic, such as queueing disciplines and dispatching policies, must adhere
+// to. By building on these interfaces, the Flow Control system can be extended and customized without modifying the
+// core controller logic.
+//
+// The primary interfaces defined here are:
+//   - `SafeQueue`: The contract for concurrent-safe queue implementations.
+//   - `ItemComparator`: The contract for policy-driven logic that defines the relative priority of items within a
+//     queue.
+package framework

pkg/epp/flowcontrol/framework/errors.go

@@ -0,0 +1,44 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package framework
+
+import (
+	"errors"
+)
+
+// `SafeQueue` Errors
+//
+// These errors relate to operations directly on a `SafeQueue` implementation. They are returned by `SafeQueue` methods
+// and might be handled or wrapped by the `ports.FlowRegistry`'s `ports.ManagedQueue` or the
+// `controller.FlowController`.
+var (
+	// ErrNilQueueItem indicates that a nil `types.QueueItemAccessor` was passed to `SafeQueue.Add()`.
+	ErrNilQueueItem = errors.New("queue item cannot be nil")
+
+	// ErrQueueEmpty indicates an attempt to perform an operation on an empty `SafeQueue` that requires one or more items
+	// (e.g., calling `SafeQueue.PeekHead()`).
+	ErrQueueEmpty = errors.New("queue is empty")
+
+	// ErrInvalidQueueItemHandle indicates that a `types.QueueItemHandle` provided to a `SafeQueue` operation (e.g.,
+	// `SafeQueue.Remove()`) is not valid for that queue, has been invalidated, or does not correspond to an actual item
+	// in the queue.
+	ErrInvalidQueueItemHandle = errors.New("invalid queue item handle")
+
+	// ErrQueueItemNotFound indicates that a `SafeQueue.Remove(handle)` operation did not find an item matching the
+	// provided, valid `types.QueueItemHandle`. This can occur if the item was removed by a concurrent operation.
+	ErrQueueItemNotFound = errors.New("queue item not found for the given handle")
+)

pkg/epp/flowcontrol/framework/mocks/mocks.go

@@ -0,0 +1,32 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package mocks
+
+import (
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/flowcontrol/framework"
+)
+
+// MockItemComparator provides a mock implementation of the `framework.ItemComparator` interface.
+type MockItemComparator struct {
+	FuncV      framework.ItemComparatorFunc
+	ScoreTypeV string
+}
+
+func (m *MockItemComparator) Func() framework.ItemComparatorFunc { return m.FuncV }
+func (m *MockItemComparator) ScoreType() string                  { return m.ScoreTypeV }
+
+var _ framework.ItemComparator = &MockItemComparator{}

pkg/epp/flowcontrol/framework/plugins/queue/README.md

@@ -0,0 +1,114 @@
+# Flow Controller Queue Plugins (`plugins/queue/`)
+
+This directory contains concrete implementations of the `framework.SafeQueue` interface. This contract defines core,
+self-contained queue data structures used by the `controller.FlowController`.
+
+## Overview
+
+The `controller.FlowController` manages requests by organizing them into queues. Each logical "flow" (e.g., a specific
+model or workload) within a given priority band has its own `ports.ManagedQueue` instance, which wraps a
+`framework.SafeQueue`. This design allows the `controller.FlowController` to apply policies at both the inter-flow
+(across different flows) and intra-flow (within a single flow's queue) levels.
+
+The `framework.SafeQueue` interface abstracts the underlying data structure and its ordering logic. This pluggable
+design allows for:
+
+- **Different Queuing Disciplines**: A basic FIFO queue ([`listqueue`](./listqueue/)) is provided, but other disciplines
+  like priority queues ([`maxminheap`](./maxminheap/)) can be used for more complex ordering requirements.
+- **Specialized Capabilities**: Policies can declare `RequiredQueueCapabilities()` (e.g., `framework.CapabilityFIFO` or
+  `framework.CapabilityPriorityConfigurable`). The `ports.FlowRegistry` pairs the policy with a queue that provides the
+  necessary capabilities.
+- **Performance Optimization**: Different queue implementations offer varying performance characteristics, which can be
+  compared using the centralized benchmark suite to select the best fit for a given workload.
+
+## Contributing a New `SafeQueue` Implementation
+
+To contribute a new queue implementation, follow these steps:
+
+1.  **Define Your Implementation**
+    - Create a new Go package in a subdirectory (e.g., `mycustomqueue/`).
+    - Implement the `framework.SafeQueue` and `types.QueueItemHandle` interfaces.
+    - Ensure all methods of `framework.SafeQueue` are goroutine-safe, typically by using a `sync.Mutex` or
+      `sync.RWMutex`.
+    - If your queue declares `framework.CapabilityPriorityConfigurable`, it MUST use the `framework.ItemComparator`
+      passed to its constructor for all internal ordering logic.
+
+2. **Register Your Queue**
+    - In an `init()` function within your queue's Go file, call `queue.MustRegisterQueue()` with a unique name and a
+      constructor function that matches the `queue.QueueConstructor` signature.
+
+3.  **Add to the Conformance Test**
+    - Add a blank import for your new package to [`conformance_test.go`](./conformance_test.go). Your queue will then be
+      automatically included in the conformance suite, which validates the `SafeQueue` contract.
+
+4.  **Documentation**
+    - Add GoDoc comments to your new queue type, explaining its behavior, capabilities, and any trade-offs.
+
+5.  **Benchmarking**
+    - You do not need to write custom benchmarks. The centralized suite in [`benchmark_test.go`](./benchmark_test.go)
+      automatically includes any new queue implementation after it is registered. This ensures all queues are compared
+      fairly under the same conditions.
+
+## Benchmarking Strategy and Results
+
+A centralized benchmark suite runs against all registered `SafeQueue` implementations to provide a consistent
+performance comparison. To run the benchmarks, use the following command:
+
+```sh
+go test -bench=. -benchmem ./pkg/epp/flowcontrol/framework/plugins/queue/...
+```
+
+### Benchmark Scenarios
+
+The suite includes the following scenarios:
+
+- **`AddRemove`**: Measures throughput of tightly coupled `Add` and `Remove` operations under high parallelism. This
+  tests the raw overhead of the data structure and its locking mechanism for simple, transactional workloads.
+- **`AddPeekRemove`**: Measures performance of a sequential `Add` -> `PeekHead` -> `Remove` loop. This simulates a
+  common consumer pattern where a single worker inspects an item before processing it.
+- **`BulkAddThenBulkRemove`**: Tests performance of adding a large batch of items and then removing them all. This can
+  reveal how the data structure's performance changes as it grows and shrinks under load.
+- **`HighContention`**: Simulates a realistic workload with multiple concurrent producers (adding items) and consumers
+  (peeking and removing items) operating on the same queue.
+
+### Latest Results
+
+*Last Updated: 2025-07-10*
+*(CPU: AMD EPYC 7B12)*
+
+| Benchmark                   | Implementation | Iterations | ns/op   | B/op  | allocs/op |
+| --------------------------- | -------------- | ---------- | ------- | ----- | --------- |
+| **AddRemove**               | `ListQueue`    | 1,889,844  | 609.0   | 224   | 5         |
+|                             | `MaxMinHeap`   | 1,660,987  | 696.7   | 184   | 4         |
+| **AddPeekRemove**           | `ListQueue`    | 3,884,938  | 298.0   | 224   | 5         |
+|                             | `MaxMinHeap`   | 1,857,448  | 615.9   | 184   | 4         |
+| **AddPeekTailRemove**       | `ListQueue`    | 3,576,487  | 308.4   | 224   | 5         |
+|                             | `MaxMinHeap`   | 2,113,134  | 535.3   | 184   | 4         |
+| **BulkAddThenBulkRemove**   | `ListQueue`    | 24,032     | 49,861  | 24801 | 698       |
+|                             | `MaxMinHeap`   | 10,000     | 108,868 | 20787 | 597       |
+| **HighContention**          | `ListQueue`    | 484,574    | 2,328   | 896   | 20        |
+|                             | `MaxMinHeap`   | 84,806     | 18,679  | 783   | 16        |
+
+### Interpretation of Results
+
+The benchmark results highlight the trade-offs between the different queue implementations based on their underlying
+data structures:
+
+- **`ListQueue`**: As a linked list, it excels in scenarios involving frequent additions or removals from either end of
+  the queue (`AddPeekRemove`, `AddPeekTailRemove`), which are O(1) operations. Its performance is less competitive in
+  high-contention and bulk scenarios, which reflects the necessary per-item memory allocation and pointer manipulation
+  overhead.
+- **`MaxMinHeap`**: As a slice-based heap, it has a lower allocation overhead per operation, making it efficient for
+  high-throughput `AddRemove` cycles. Peeking and removing items involves maintaining the heap property, which has an
+  O(log n) cost, making individual peek operations slower than `ListQueue`.
+
+**Choosing a Queue:**
+
+The data suggests the following guidance:
+- For simple **FIFO** workloads where the primary operations are consuming from the head, `ListQueue` is a strong and
+  simple choice.
+- For workloads requiring **priority-based ordering** or those that are sensitive to allocation overhead under high
+  contention, `MaxMinHeap` is likely the more suitable option.
+
+These benchmarks provide a baseline for performance. The best choice for a specific use case will depend on the expected
+workload patterns.