Add subsetting logic for epp #981

rlakhtakia · 2025-06-13T09:20:19Z

Add subsetting filter to ensure EPP only selects from the list of endpoints passed in through request metadata.

Changes:

Filter logic + unit test + integration test
Update request type to pass in metadata context
Update scheduler profiles to use filter

netlify · 2025-06-13T09:20:24Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`4364512`
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/685c58d0b17b570008c0605e
😎 Deploy Preview	https://deploy-preview-981--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

k8s-ci-robot · 2025-06-13T09:20:29Z

Hi @rlakhtakia. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

nirrozenbaum

envoy specifics should be only in the server.go.
rest of the code should work with go general structs like maps

pkg/epp/scheduling/types/types.go

cmd/epp/runner/runner.go

pkg/epp/scheduling/types/types.go

pkg/epp/scheduling/framework/plugins/filter/subsetting_filter.go

nirrozenbaum · 2025-06-16T18:21:35Z

/ok-to-test

kfswain

just a couple comments to future proof us! otherwise lgtm

pkg/epp/handlers/server.go

pkg/epp/util/request/metadata.go

kfswain · 2025-06-17T17:17:35Z

/retest

rlakhtakia · 2025-06-17T18:10:55Z

/retest

liu-cong

just a few nits

pkg/epp/scheduling/framework/plugins/filter/subsetting_filter.go

test/integration/util.go

test/integration/epp/hermetic_test.go

pkg/epp/scheduling/framework/plugins/filter/subsetting_filter.go

rlakhtakia · 2025-06-23T16:50:52Z

/retest

nirrozenbaum · 2025-06-23T18:04:20Z

To me it makes sense that any immutable features run strictly before or after any user configured code. So that works for Filters for now, but may not always be the case.

++. agreed.
Subset Filter works for now, but this may not always be the case.
anyway, I didn't intend to block this PR, just to raise a point to think about.
let's make progress with merging this one as soon as it passes the tests and we can think about this more as we make progress.

pkg/epp/scheduling/framework/plugins/filter/subsetting_filter.go

ahg-g · 2025-06-23T18:21:10Z

@rlakhtakia pls rebase and run unit tests locally, this should allow you to find compile time problems earlier.

nirrozenbaum · 2025-06-25T08:24:17Z

I went over the PR. overall it LGTM if we decide to keep it as filter.

one additional concern about the earlier discussion on this thread -
in case this is implemented via filter and we have multiple scheduling profiles - this filter has to run in each.
so if for example we have 4 profiles, 10000 candidate pods, and we specify in the subset key a list of 10 pods to choose from - we will need to filter the 10 pods out of the 10000 candidates 4 times.

To me it makes sense that any immutable features run strictly before or after any user configured code. So that works for Filters for now, but may not always be the case.

so in addition to the above comment @kfswain wrote, this works fine in current design as long as we have only one profile (not the case in llm-d).
the alternative is to implement it here (replace ds.PodGetAll with ds.PodList(predicate):

gateway-api-inference-extension/pkg/epp/requestcontrol/director.go

Lines 141 to 142 in fe5dea3

    
           candidatePods := schedulingtypes.ToSchedulerPodMetrics(d.datastore.PodGetAll()) 
        
           results, err := d.scheduler.Schedule(ctx, reqCtx.SchedulingRequest, candidatePods)

and then candidate pods are filtered only once before scheduling, no matter how many profiles we have.

@ahg-g @kfswain leaving the final stamp for you to decide if we want to merge or not.

ahg-g

Thanks! One comment on how to deal with cases where the metadata exists but the list is empty, otherwise looks great.

ahg-g · 2025-06-25T15:13:47Z

cmd/epp/runner/runner.go

 		kvCacheScorerWeight := envutil.GetEnvInt("KV_CACHE_SCORE_WEIGHT", scorer.DefaultKVCacheScorerWeight, setupLog)

 		schedulerProfile := framework.NewSchedulerProfile().
+			WithFilters(filter.NewSubsetFilter()).


This is fine for now, but please open an issue to allow configuring this across profiles irrespective of the source of the configuration (see the discussion we had on the issue)

pkg/epp/scheduling/framework/plugins/filter/subsetting_filter.go

ahg-g · 2025-06-25T19:36:26Z

pkg/epp/scheduling/framework/plugins/filter/subsetting_filter.go

+	if !found {
+		return pods
+	} else if len(endpointSubsetList) == 0 {
+		return pods


Need to return an empty list here, not all pods, meaning all pods are filtered.

fixed now, and added a testcase to verify this.

ahg-g · 2025-06-25T20:22:38Z

@nirrozenbaum agree on the need for a followup, my thinking is to have a "framework" for configuring mandatory plugins via code (i.e., this is no a user facing api). An initial cut would be as simple as a list of filters that get prepended to all profiles. Since this is hardcoded, we can evolve it slowly and by reacting to new needs, it doesn't need to be fully fleshed out from the beginning since it is not user facing.

ahg-g · 2025-06-25T20:27:35Z

Created #1068 as a follow up

ahg-g · 2025-06-25T20:28:52Z

/lgtm
/approve

Congrats @rlakhtakia on your first inference gateway PR!

k8s-ci-robot · 2025-06-25T20:28:59Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahg-g, rlakhtakia

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ahg-g]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Add APIs for the instantiated plugins to the EPP Handle (kubernetes-sigs#1039) * Added plugin instance APIs to plugins.Handle Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * An implementation of the new plugins.Handle APIs Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Moved all configuration loading code to new package Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Updates due to new and moved APIs Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Cleanup of old configuration loading code Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> --------- Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> chore(deps): bump the kubernetes group with 6 updates (kubernetes-sigs#1050) Bumps the kubernetes group with 6 updates: | Package | From | To | | --- | --- | --- | | [k8s.io/api](https://github.com/kubernetes/api) | `0.33.1` | `0.33.2` | | [k8s.io/apiextensions-apiserver](https://github.com/kubernetes/apiextensions-apiserver) | `0.33.1` | `0.33.2` | | [k8s.io/apimachinery](https://github.com/kubernetes/apimachinery) | `0.33.1` | `0.33.2` | | [k8s.io/client-go](https://github.com/kubernetes/client-go) | `0.33.1` | `0.33.2` | | [k8s.io/code-generator](https://github.com/kubernetes/code-generator) | `0.33.1` | `0.33.2` | | [k8s.io/component-base](https://github.com/kubernetes/component-base) | `0.33.1` | `0.33.2` | Updates `k8s.io/api` from 0.33.1 to 0.33.2 - [Commits](kubernetes/api@v0.33.1...v0.33.2) Updates `k8s.io/apiextensions-apiserver` from 0.33.1 to 0.33.2 - [Release notes](https://github.com/kubernetes/apiextensions-apiserver/releases) - [Commits](kubernetes/apiextensions-apiserver@v0.33.1...v0.33.2) Updates `k8s.io/apimachinery` from 0.33.1 to 0.33.2 - [Commits](kubernetes/apimachinery@v0.33.1...v0.33.2) Updates `k8s.io/client-go` from 0.33.1 to 0.33.2 - [Changelog](https://github.com/kubernetes/client-go/blob/master/CHANGELOG.md) - [Commits](kubernetes/client-go@v0.33.1...v0.33.2) Updates `k8s.io/code-generator` from 0.33.1 to 0.33.2 - [Commits](kubernetes/code-generator@v0.33.1...v0.33.2) Updates `k8s.io/component-base` from 0.33.1 to 0.33.2 - [Commits](kubernetes/component-base@v0.33.1...v0.33.2) --- updated-dependencies: - dependency-name: k8s.io/api dependency-version: 0.33.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: kubernetes - dependency-name: k8s.io/apiextensions-apiserver dependency-version: 0.33.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: kubernetes - dependency-name: k8s.io/apimachinery dependency-version: 0.33.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: kubernetes - dependency-name: k8s.io/client-go dependency-version: 0.33.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: kubernetes - dependency-name: k8s.io/code-generator dependency-version: 0.33.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: kubernetes - dependency-name: k8s.io/component-base dependency-version: 0.33.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: kubernetes ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> remove datastore dependency from the scheduler (kubernetes-sigs#1049) * remove datastore dependency from the scheduler Signed-off-by: Nir Rozenbaum <nirro@il.ibm.com> * added back comments on snapshotting pods from datastore before calling schedule Signed-off-by: Nir Rozenbaum <nirro@il.ibm.com> * removed fake datastore from conformance scheduler test Signed-off-by: Nir Rozenbaum <nirro@il.ibm.com> --------- Signed-off-by: Nir Rozenbaum <nirro@il.ibm.com> Add subsetting logic for epp (kubernetes-sigs#981) feat: Added a factory function for the DecisionTree filter (kubernetes-sigs#1053) * Added a factory function for the DecisionTreeFilter Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Added tests of the factory function of the DecisionTreeFilter Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Registered the factory function of the DecisionTreeFilter Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Refactored the configuration loading Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> --------- Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> Adding pprof endpoints to metrics port (kubernetes-sigs#1069) feat: Add a context.Context to the plugins.HAndle interface (kubernetes-sigs#1076) * Added a context.Context to the plugins.Handle interface Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Changes due to changes in internal APIs Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Changes to tests due to changes in internal APIs Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> --------- Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> convert subset filter from a plugin to logic in director (kubernetes-sigs#1088) * convert subset filter from a plugin to logic in director Signed-off-by: Nir Rozenbaum <nirro@il.ibm.com> * replace interface{} with any Signed-off-by: Nir Rozenbaum <nirro@il.ibm.com> * make linter happy Signed-off-by: Nir Rozenbaum <nirro@il.ibm.com> * address code review comments Signed-off-by: Nir Rozenbaum <nirro@il.ibm.com> --------- Signed-off-by: Nir Rozenbaum <nirro@il.ibm.com> chore(deps): bump golang.org/x/sync from 0.14.0 to 0.15.0 (kubernetes-sigs#1096) Bumps [golang.org/x/sync](https://github.com/golang/sync) from 0.14.0 to 0.15.0. - [Commits](golang/sync@v0.14.0...v0.15.0) --- updated-dependencies: - dependency-name: golang.org/x/sync dependency-version: 0.15.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Introduce plugins.TypedName to be used for Plugin base implementation (kubernetes-sigs#1086) * introduce TypedName to reduce boilerplate, modify plugins Signed-off-by: Etai Lev Ran <elevran@gmail.com> * implement GetTypedName() Signed-off-by: Etai Lev Ran <elevran@gmail.com> * Remove Type() and Name() from Plugin interface Signed-off-by: Etai Lev Ran <elevran@gmail.com> * use TypedName as private field, not embedded Signed-off-by: Etai Lev Ran <elevran@gmail.com> --------- Signed-off-by: Etai Lev Ran <elevran@gmail.com> move the conversion from pod metrics to scheduler pod representation one level up (kubernetes-sigs#1104) * move the converstion from pod metrics to scheduler pod representation one level up Signed-off-by: Nir Rozenbaum <nirro@il.ibm.com> * minor change in helper func Signed-off-by: Nir Rozenbaum <nirro@il.ibm.com> --------- Signed-off-by: Nir Rozenbaum <nirro@il.ibm.com> handle picking multiple destinations in scheduling layer (kubernetes-sigs#1059) * implement multiple destination as the output of the scheduler Signed-off-by: Nir Rozenbaum <nirro@il.ibm.com> * updated max score picker unit tests to cover multiple pods Signed-off-by: Nir Rozenbaum <nirro@il.ibm.com> * imports Signed-off-by: Nir Rozenbaum <nirro@il.ibm.com> * unit-test fix Signed-off-by: Nir Rozenbaum <nirro@il.ibm.com> --------- Signed-off-by: Nir Rozenbaum <nirro@il.ibm.com> refactor: 🔨 use the more explicit singular form (kubernetes-sigs#1129)

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 13, 2025

k8s-ci-robot requested review from ahg-g and nirrozenbaum June 13, 2025 09:20

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jun 13, 2025

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jun 13, 2025

nirrozenbaum reviewed Jun 13, 2025

View reviewed changes

kfswain reviewed Jun 13, 2025

View reviewed changes

pkg/epp/scheduling/types/types.go Outdated Show resolved Hide resolved

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 13, 2025

rlakhtakia force-pushed the filter branch from 9660d65 to edba0ce Compare June 13, 2025 22:34

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 13, 2025

rlakhtakia force-pushed the filter branch from edba0ce to 74d12ca Compare June 13, 2025 22:45

nirrozenbaum reviewed Jun 15, 2025

View reviewed changes

cmd/epp/runner/runner.go Outdated Show resolved Hide resolved

nirrozenbaum reviewed Jun 15, 2025

View reviewed changes

pkg/epp/scheduling/types/types.go Show resolved Hide resolved

nirrozenbaum reviewed Jun 15, 2025

View reviewed changes

pkg/epp/scheduling/framework/plugins/filter/subsetting_filter.go Show resolved Hide resolved

rlakhtakia force-pushed the filter branch from 74d12ca to b42618a Compare June 16, 2025 16:53

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 16, 2025

kfswain reviewed Jun 16, 2025

View reviewed changes

pkg/epp/handlers/server.go Outdated Show resolved Hide resolved

pkg/epp/util/request/metadata.go Outdated Show resolved Hide resolved

rlakhtakia force-pushed the filter branch from b42618a to 784fe3a Compare June 17, 2025 17:08

rlakhtakia force-pushed the filter branch from 784fe3a to d35c57d Compare June 17, 2025 18:10

liu-cong reviewed Jun 17, 2025

View reviewed changes

pkg/epp/scheduling/framework/plugins/filter/subsetting_filter.go Show resolved Hide resolved

test/integration/util.go Outdated Show resolved Hide resolved

test/integration/epp/hermetic_test.go Outdated Show resolved Hide resolved

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 18, 2025

rlakhtakia force-pushed the filter branch from d35c57d to 907d51f Compare June 18, 2025 23:39

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 18, 2025

nayihz reviewed Jun 19, 2025

View reviewed changes

pkg/epp/scheduling/framework/plugins/filter/subsetting_filter.go Outdated Show resolved Hide resolved

nirrozenbaum reviewed Jun 23, 2025

View reviewed changes

pkg/epp/scheduling/framework/plugins/filter/subsetting_filter.go Outdated Show resolved Hide resolved

rlakhtakia force-pushed the filter branch from 659ccce to 7928bdd Compare June 24, 2025 22:15

ahg-g reviewed Jun 25, 2025

View reviewed changes

rlakhtakia force-pushed the filter branch from 7928bdd to 2b96469 Compare June 25, 2025 19:22

ahg-g reviewed Jun 25, 2025

View reviewed changes

Add subsetting logic for epp

4364512

rlakhtakia force-pushed the filter branch from 2b96469 to 4364512 Compare June 25, 2025 20:15

ahg-g mentioned this pull request Jun 25, 2025

Align on how to configure mandatory plugins (and whether they need to be modeled as plugins in the first place) #1068

Open

k8s-ci-robot assigned ahg-g Jun 25, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 25, 2025

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 25, 2025

k8s-ci-robot merged commit 6b82b89 into kubernetes-sigs:main Jun 25, 2025
9 checks passed

rlakhtakia deleted the filter branch June 26, 2025 07:36

rlakhtakia added a commit to rlakhtakia/gateway-api-inference-extension that referenced this pull request Jun 26, 2025

Add subsetting logic for epp (kubernetes-sigs#981)

8810da4

shmuelk pushed a commit to shmuelk/gateway-api-inference-extension that referenced this pull request Jun 26, 2025

Add subsetting logic for epp (kubernetes-sigs#981)

0f73b10

rlakhtakia added a commit to rlakhtakia/gateway-api-inference-extension that referenced this pull request Jun 26, 2025

Add subsetting logic for epp (kubernetes-sigs#981)

cc1e3c6

EyalPazz pushed a commit to EyalPazz/gateway-api-inference-extension that referenced this pull request Jul 9, 2025

Add subsetting logic for epp (kubernetes-sigs#981)

ca85da0

BenjaminBraunDev pushed a commit to BenjaminBraunDev/gateway-api-inference-extension that referenced this pull request Aug 12, 2025

Add subsetting logic for epp (kubernetes-sigs#981)

a05afc3

Add subsetting logic for epp #981

Add subsetting logic for epp #981

Uh oh!

Conversation

rlakhtakia commented Jun 13, 2025

Uh oh!

netlify bot commented Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for gateway-api-inference-extension ready!

Uh oh!

k8s-ci-robot commented Jun 13, 2025

Uh oh!

nirrozenbaum left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nirrozenbaum commented Jun 16, 2025

Uh oh!

kfswain left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kfswain commented Jun 17, 2025

Uh oh!

rlakhtakia commented Jun 17, 2025

Uh oh!

liu-cong left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rlakhtakia commented Jun 23, 2025

Uh oh!

nirrozenbaum commented Jun 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

ahg-g commented Jun 23, 2025

Uh oh!

nirrozenbaum commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ahg-g left a comment

Choose a reason for hiding this comment

Uh oh!

ahg-g Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ahg-g Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

rlakhtakia Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

ahg-g commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ahg-g commented Jun 25, 2025

Uh oh!

ahg-g commented Jun 25, 2025

Uh oh!

k8s-ci-robot commented Jun 25, 2025

Uh oh!

Uh oh!

Uh oh!

netlify bot commented Jun 13, 2025 •

edited

Loading

nirrozenbaum left a comment •

edited

Loading

nirrozenbaum commented Jun 23, 2025 •

edited

Loading

nirrozenbaum commented Jun 25, 2025 •

edited

Loading

ahg-g commented Jun 25, 2025 •

edited

Loading