feat(conformance): Add HTTPRouteMultipleRulesDifferentPools test #834

SinaChavoshi · 2025-05-14T22:40:02Z

Description

This PR introduces a new conformance test, HTTPRouteMultipleRulesDifferentPools, which validates the setup with one Gateway and one HTTPRoute to successfully route traffic to multiple, distinct InferencePool backends.

local run results: ( Ran on commit daeb8e6)

go test -v ./conformance -args -debug     -gateway-class gke-l7-regional-external-managed     -cleanup-base-resources=false     -run-test HTTPRouteMultipleRulesDifferentPools
=== RUN   TestConformance
...
    apply.go:279: 2025-06-17T00:17:03.69078318Z: Deleting pool-primary-epp Service
=== NAME  TestConformance
    suite.go:451: 2025-06-17T00:17:03.869842294Z: Sleeping 0s for test isolation
=== RUN   TestConformance/InferencePoolResolvedRefsCondition
    conformance.go:68: Skipping InferencePoolResolvedRefsCondition: test explicitly skipped
--- PASS: TestConformance (70.46s)
    --- SKIP: TestConformance/HTTPRouteInvalidInferencePoolRef (0.00s)
    --- SKIP: TestConformance/InferencePoolAccepted (0.00s)
    --- PASS: TestConformance/HTTPRouteMultipleRulesDifferentPools (66.56s)
        --- PASS: TestConformance/HTTPRouteMultipleRulesDifferentPools/Wait_for_resources_to_be_accepted (34.50s)
        --- PASS: TestConformance/HTTPRouteMultipleRulesDifferentPools/Traffic_should_be_routed_to_the_correct_pool_based_on_path (28.87s)
            --- PASS: TestConformance/HTTPRouteMultipleRulesDifferentPools/Traffic_should_be_routed_to_the_correct_pool_based_on_path/request_to_primary_pool (28.20s)
            --- PASS: TestConformance/HTTPRouteMultipleRulesDifferentPools/Traffic_should_be_routed_to_the_correct_pool_based_on_path/request_to_secondary_pool (0.24s)
    --- SKIP: TestConformance/InferencePoolResolvedRefsCondition (0.00s)
PASS
ok      sigs.k8s.io/gateway-api-inference-extension/conformance 70.674s

k8s-ci-robot · 2025-05-14T22:40:12Z

Hi @SinaChavoshi. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

netlify · 2025-05-14T22:40:17Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`81562a2`
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/6851a00997ab93000885f394
😎 Deploy Preview	https://deploy-preview-834--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

conformance/tests/basic/inferencepool_multiple_rules_different_pools.yaml

spencerhance · 2025-05-16T18:16:12Z

/cc

ahg-g · 2025-05-20T15:15:31Z

/ok-to-test

conformance/tests/basic/inferencepool_multiple_rules_different_pools.yaml

SinaChavoshi · 2025-05-28T18:43:09Z

@robscott @danehans Once #866 is merged in I will update this PR to use the same helper methods for route accepted and pool accepted. For now leaving the PR as is to avoid copying code across multiple PRs.

conformance/resources/manifests/manifests.yaml

conformance/tests/basic/inferencepool_multiple_rules_different_pools.yaml

conformance/utils/traffic/traffic.go

robscott

Thanks @SinaChavoshi, this is great!

/lgtm

conformance/utils/kubernetes/helpers.go

conformance/tests/basic/inferencepool_multiple_rules_different_pools.go

danehans · 2025-06-17T20:39:22Z

/lgtm
/approve

k8s-ci-robot · 2025-06-17T20:39:31Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danehans, robscott, SinaChavoshi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [danehans]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…ernetes-sigs#834) * copy of accepted inference pool test to start from. * add yaml file for the test * update time out * update the yaml file to add port 9002 * read timeout config from local repo * remove excess comments * correct spelling for scenarios * check route condition on RouteConditionResolvedRefs * remove empty lines in yaml * set optional/defaulted fields as unspecified * fix timeout * fix boilerplate header * change varialbe names to use primary secondary consistently. * remove extra comments * factor out common code * Add actual http traffic validation using echo-basic * remove extra comments from manifest * remove modifiedTimeoutConfig.HTTPRouteMustHaveCondition per review comment. * intermediate update * fix the test run * factor out common code * move epp def to shared manifest * remove extra comments * revert back to two epps * add to do for epp image * switch to GeneralMustHaveConditionTimeout * undo gateway version changes * remove unused HTTPRouteMustHaveConditions * update doc string for GetPod * update docstring * Remove resource type from names in manifests. * remove type from name * remove health check * add todo for combining getpod methods

…e it easier to add plugins (#881) * configuration implementation (after rebase...) Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Moved plugin registry back to pkg/epp/plugins Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Removed unneeded 'forced imports' of scorers Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Changed 'profilepicker' to 'profilehandler' in new and old code Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Pass the configured SchedulingProfiles to LoadSchedulerConfig Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Ensure that both the configText and configFile flags are not specified Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Load RequestControl plugins from the configuration Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Register all plugin factories Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Review fixes Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Reverted unneeded change Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Updates from review comments Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Added a stub interface for plugins to get data from the EPP Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Added a temporary implementation of plugins.Handle Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Added pluginName and plugins.Handle to plugin factory interface Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Updated plugin factory signatures to reflect new API Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Updated plugin instantiation to reflect new API Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Updated plugin instantiation to reflect new API Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Updated tests to reflect new API Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Do not rename the imported package Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Only upper layer of code should log errors Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Only pass what is needed to instantiate the plugins Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Review updates Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Review update Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Review update. Make more clear that the code only checks for already defined names Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * fixed e2e doc in makefile (does not require GPUs) (#976) Signed-off-by: Nir Rozenbaum <nirro@il.ibm.com> * API: Adds 5xx Status Code for Invalid ExtRef (#991) Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io> * feat(conformance): Add test for invalid EPP service reference (#959) * fix boilerplate header * add tests for InferencePoolInvalidEPPService * change to expect error on httproute refcond * moved the creation of the context to main.go. (#995) this is useful when writing a different main like llm-d, allowing to propogate the same context to the whole system. Signed-off-by: Nir Rozenbaum <nirro@il.ibm.com> * fix dead links (#989) * feat: add health check for epp cluster (#966) * feat: add health check for epp cluster Signed-off-by: zhengkezhou1 <madzhou1@gmail.com> * remove tls Signed-off-by: zhengkezhou1 <madzhou1@gmail.com> * don't use tls Signed-off-by: zhengkezhou1 <madzhou1@gmail.com> * health checking flag Signed-off-by: zhengkezhou1 <madzhou1@gmail.com> * fix import Signed-off-by: zhengkezhou1 <madzhou1@gmail.com> * add tls options Signed-off-by: zhengkezhou1 <madzhou1@gmail.com> --------- Signed-off-by: zhengkezhou1 <madzhou1@gmail.com> * Server unit test and utility to help with such tests (#820) Signed-off-by: Ira <IRAR@il.ibm.com> * Update dynamic-lora-sidecar to expose metrics to track loaded adapters (#980) * Add a metrics to track loaded adapters * Update the sample manifests * Add explanation of metrics from dyanmic LoRA adapter sidecar * Add explanation of metrics from dyanmic LoRA adapter sidecar (take 2) * Update metrics.md based on feedback * refactor: Replace prefix cache structure with golang-lru (#928) * refactor: Replace prefix cache structure with golang-lru Signed-off-by: Kfir Toledo <kfir.toledo@ibm.com> Co-authored-by: Maroon Ayoub <maroon.ayoub@ibm.com> * fix: rename prefix scorer parameters and convert test to benchmark test Signed-off-by: Kfir Toledo <kfir.toledo@ibm.com> * feat: Add per server LRU capacity Signed-off-by: Kfir Toledo <kfir.toledo@ibm.com> * fix: Fix typos and error handle Signed-off-by: Kfir Toledo <kfir.toledo@ibm.com> * fix: add safety check for LRUCapacityPerServer Signed-off-by: Kfir Toledo <kfir.toledo@ibm.com> --------- Signed-off-by: Kfir Toledo <kfir.toledo@ibm.com> Co-authored-by: Maroon Ayoub <maroon.ayoub@ibm.com> * feat(conformance): Add HTTPRouteMultipleRulesDifferentPools test (#834) * copy of accepted inference pool test to start from. * add yaml file for the test * update time out * update the yaml file to add port 9002 * read timeout config from local repo * remove excess comments * correct spelling for scenarios * check route condition on RouteConditionResolvedRefs * remove empty lines in yaml * set optional/defaulted fields as unspecified * fix timeout * fix boilerplate header * change varialbe names to use primary secondary consistently. * remove extra comments * factor out common code * Add actual http traffic validation using echo-basic * remove extra comments from manifest * remove modifiedTimeoutConfig.HTTPRouteMustHaveCondition per review comment. * intermediate update * fix the test run * factor out common code * move epp def to shared manifest * remove extra comments * revert back to two epps * add to do for epp image * switch to GeneralMustHaveConditionTimeout * undo gateway version changes * remove unused HTTPRouteMustHaveConditions * update doc string for GetPod * update docstring * Remove resource type from names in manifests. * remove type from name * remove health check * add todo for combining getpod methods * configuration implementation (after rebase...) Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * After review, made code more obvious Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> * Fixed merge issues Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> --------- Signed-off-by: Shmuel Kallner <kallner@il.ibm.com> Signed-off-by: Nir Rozenbaum <nirro@il.ibm.com> Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io> Signed-off-by: zhengkezhou1 <madzhou1@gmail.com> Signed-off-by: Ira <IRAR@il.ibm.com> Signed-off-by: Kfir Toledo <kfir.toledo@ibm.com> Co-authored-by: Nir Rozenbaum <nirro@il.ibm.com> Co-authored-by: Daneyon Hansen <daneyon.hansen@solo.io> Co-authored-by: sina chavoshi <chavoshi@google.com> Co-authored-by: Xudong Wang <68834160+caozhuozi@users.noreply.github.com> Co-authored-by: Zhengke Zhou <madzhou1@gmail.com> Co-authored-by: Ira Rosen <irar@il.ibm.com> Co-authored-by: Shotaro Kohama <khmshtr28@gmail.com> Co-authored-by: Kfir Toledo <kfir.toledo@gmail.com> Co-authored-by: Maroon Ayoub <maroon.ayoub@ibm.com>

SinaChavoshi added 6 commits May 14, 2025 20:30

copy of accepted inference pool test to start from.

87d1756

add yaml file for the test

a9dcb62

update time out

c36e5f8

update the yaml file to add port 9002

aaf23e2

read timeout config from local repo

2a67e7d

remove excess comments

8ac7d72

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label May 14, 2025

k8s-ci-robot requested review from ahg-g and kfswain May 14, 2025 22:40

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label May 14, 2025

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label May 14, 2025

nirrozenbaum reviewed May 15, 2025

View reviewed changes

conformance/tests/basic/inferencepool_multiple_rules_different_pools.yaml Outdated Show resolved Hide resolved

k8s-ci-robot requested a review from spencerhance May 16, 2025 18:16

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 20, 2025

SinaChavoshi added 2 commits May 20, 2025 17:02

correct spelling for scenarios

66f325d

check route condition on RouteConditionResolvedRefs

fd0fa10

danehans reviewed May 27, 2025

View reviewed changes

SinaChavoshi added 3 commits May 27, 2025 17:29

Merge branch 'main' into inf_pool_tests_11

59b1f9c

remove empty lines in yaml

8a078dc

set optional/defaulted fields as unspecified

c55a803

SinaChavoshi mentioned this pull request May 27, 2025

feat(conformance): Add HTTPRouteMultipleGatewaysDifferentPools test #838

Merged

Merge branch 'main' into inf_pool_tests_11

c6bacb4

SinaChavoshi added 2 commits May 28, 2025 21:44

Merge branch 'main' into inf_pool_tests_11

271b7a8

fix timeout

c92a402

SinaChavoshi added 4 commits June 16, 2025 20:09

undo gateway version changes

112d878

remove unused HTTPRouteMustHaveConditions

120feb3

update doc string for GetPod

8214cdf

update docstring

76cb644

danehans requested a review from robscott June 16, 2025 21:46

danehans reviewed Jun 16, 2025

View reviewed changes

SinaChavoshi added 3 commits June 16, 2025 23:58

Remove resource type from names in manifests.

5dca67b

remove type from name

f2c0602

remove health check

daeb8e6

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jun 17, 2025

robscott approved these changes Jun 17, 2025

View reviewed changes

k8s-ci-robot assigned robscott Jun 17, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 17, 2025

robscott reviewed Jun 17, 2025

View reviewed changes

conformance/utils/kubernetes/helpers.go Show resolved Hide resolved

Merge remote-tracking branch 'upstream/main' into inf_pool_tests_11

f1ab79f

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 17, 2025

SinaChavoshi mentioned this pull request Jun 17, 2025

refactor:(conformance): combine methods for getting pods in helper method. #1003

Closed

add todo for combining getpod methods

81562a2

danehans reviewed Jun 17, 2025

View reviewed changes

conformance/tests/basic/inferencepool_multiple_rules_different_pools.go Show resolved Hide resolved

k8s-ci-robot assigned danehans Jun 17, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 17, 2025

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 17, 2025

k8s-ci-robot merged commit 68c73c0 into kubernetes-sigs:main Jun 17, 2025
8 of 9 checks passed

SinaChavoshi mentioned this pull request Jun 17, 2025

conformance: Factor out the common constants in the resources pkg. #1006

Closed

zetxqx mentioned this pull request Jun 20, 2025

Conformance Tests for v0.5 Release #977

Closed

12 tasks

feat(conformance): Add HTTPRouteMultipleRulesDifferentPools test #834

feat(conformance): Add HTTPRouteMultipleRulesDifferentPools test #834

Uh oh!

Conversation

SinaChavoshi commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

k8s-ci-robot commented May 14, 2025

Uh oh!

netlify bot commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for gateway-api-inference-extension ready!

Uh oh!

Uh oh!

spencerhance commented May 16, 2025

Uh oh!

ahg-g commented May 20, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SinaChavoshi commented May 28, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

robscott left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

danehans commented Jun 17, 2025

Uh oh!

k8s-ci-robot commented Jun 17, 2025

Uh oh!

Uh oh!

Uh oh!

SinaChavoshi commented May 14, 2025 •

edited

Loading

netlify bot commented May 14, 2025 •

edited

Loading