Refactor EPP to remove -poolName requirement and support multiple pools per namespace

[SinaChavoshi]

Summary

The EPP (End Point Picker) application currently requires a -poolName command-line argument at startup, which ties a single EPP process to a single InferencePool resource. This prevents a simpler architecture where one shared EPP deployment could serve all InferencePools within a given namespace.

Current Behavior
As discovered while debugging the HTTPRouteMultipleRulesDifferentPools (see #834 ) conformance test, the EPP application will not start without the -poolName flag. The pod logs show a fatal error on startup:

{"error": "required \"poolName\" flag not set"}

This requirement is enforced by the validateFlags function in cmd/main.go. Additionally the EPP requires a health check component that is currently 1-1 with the an inference pool.

Impact

This design has two main impacts:

• It prevents users from deploying a single, shared EPP instance to handle routing for an entire namespace of InferencePools.

• It complicates the conformance test setup. For example, the HTTPRouteMultipleRulesDifferentPools test requires two separate EPP Deployments and Services to be defined in its manifest because it tests two distinct InferencePools. This adds boilerplate and prevents us from defining a single shared EPP in the base manifests.yaml.

Proposed Solution & Discussion

Should we refactor the EPP to make the -poolName flag optional?

If the -poolName flag were omitted, the EPP could be responsible for discovering and managing all InferencePool resources that exist within the namespace specified by the required -poolNamespace flag.

This change would:

• Enable a simpler EPP deployment model.
• Allow the conformance suite to use a single shared EPP, significantly simplifying test manifests.

Opening this issue to track the discussion on whether this change aligns with the project's goals.

[robscott]

Thanks for raising this @SinaChavoshi! A follow up question here is what the behavior would be if we didn't require a 1:1 mapping between EPP and InferencePool.

Some options I can think of:

1. We add a separate flag that allows for a list of InferencePools to be specified
2. EPP supports all InferencePools within the same namespace by default

[ahg-g]

Related issue: #145

[ahg-g]

I think it is useful to have epp support more than one InferencePool for small scale deployments and testing, although we need to do some good refactoring to support this in EPP.

This will open the door to both some interesting features regarding cross pool scheduling, and others that we may not consider as a good practice.

EPP is a single point of failure in the request path, and sharding based on pool will continue to be a good practice to support high availability.

[nirrozenbaum]

I think this is important issue that was raised several times by newcomers.
Generally speaking, the mental model of GIE is having EPP <-> base model relationship of 1:1.
This model brings to table the obvious question of how to manage multiple pools, cause there are some obvious use cases where one would want to have more than one base model.

there are several alternatives to address this issue:

1. having one EPP that can support multiple pools.
2. having another layer BEFORE the EPP ext-proc that directs the incoming request based on the specified model param to the right pool (kind of similar to BBR that adds a header based on the model param), and keep EPP per base model.
3. change the mental model to support more than one base model.

I'm assuming we would like to have the ability to work with multiple pools even if point (3) is considered, cause we might want to have multiple pools from different reasons, like sharding.

so now the question is whether we think the right approach would be (1) or (2) or some hybrid approach of the two.
in approach (1) we need to keep in mind that it's possible that pool1 is very busy while pool2 is not, and the performance of pool2 might be tremendously affected by pool1's load, and therefore I'm not sure it's the desired solution.

I do have a plan to experiment with some initial design I've started for approach (2), but this will require more time due to other tasks I'm currently working on.

[danehans]

xref #252 for providing a more flexible approach to InferencePool assignment.

[danehans]

The last time we discussed this issue, #735 was one of the outcomes.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale