feat(llm-routing): route the LLM endpoint by model, with API keys

[teemow]

What does this PR do?

#603: the LLM listener routes by model, served models get a way onto it, and the endpoint can be published outside the cluster behind API keys, with every request metered.

• Models on the listener. Every AgentgatewayModel attaches to the data plane Gateway's LLM listener directly: parentRef Gateway, sectionName: llm.

  • The listener admits AgentgatewayModels only. The controller refuses a model on a listener whose allowedRoutes.kinds do not name the kind, and no HTTPRoute, the MCP catch-all included, attaches there.
  • Listener-attached models match the exact serving endpoints (/v1/chat/completions, /v1/responses, /v1/messages, /v1/messages/count_tokens, /v1/models, …). The request's model picks the model; GET /v1/models lists the public ones, and a name no model takes answers 404 model_not_found.
  • No path prefix and no in-cluster HTTPRoute. A model-serving HTTPRoute rule strips its matched PathPrefix as a serving prefix. With /v1 that left the router paths it could not classify: requests passed through unparsed and recorded no token or cost metrics. The Gateway-scoped backend.ai.routes map, whose "*": Passthrough overrode the router's classification, is removed too, since no AI backend remains on the LLM path.
  • ModelConfigs keep http://<gateway>.<namespace>.svc:<port> as their baseUrl, so no ModelConfig needs migrating.

• llmRouting.models replaces llmRouting.backend: one AgentgatewayModel per entry (name, provider — an agentgateway managed provider —, match, visibility, baseURL).

  • The default is anthropic: provider Anthropic, baseURL: https://api.anthropic.com/v1, match: claude-*, credential-free like the AgentgatewayBackend it replaces.
  • The baseURL is required: the upstream path is the baseURL's path plus the format's suffix, and a managed provider without one sends Anthropic /messages, which answers 404.
  • The router takes the first model in name order whose match takes the name, so the default is a pattern, not *.

• Served models (kserve: one AgentgatewayModel per served model on the LLM endpoint model-manager#145): the platform release renders the LLM endpoint document, a ConfigMap <release>-llm-endpoint labelled agent-platform.giantswarm.io/llm-endpoint: "true", key llm-endpoint.yaml, kind LLMEndpoint:

  • spec.parentRefs: the listener, plus the external route;
  • spec.endpoint: the listener's in-cluster URL;
  • spec.externalEndpoint: with llmRouting.external.

  With it come model-manager's Role on agentgatewaymodels in the release namespace (nothing else) and the data plane's egress to the served workloads (both flavours). All three render with llmRouting on and model-manager serving: this release's serving slice, a static kserve backend, or cluster-manager on (a GPU node pool's slice is a second release of this chart with model-manager off, fix(model-manager): read served models wherever model-manager serves #681). The discovery ConfigMap carries no spec.llmEndpoint.

• llmRouting.external (off by default): the endpoint on https://<hostPrefix>.<global.domain> behind the installation's API keys. The route <release>-llm-external has the model router as backend and matches / under its hostname; the hostname outranks the catch-all. The AgentgatewayPolicy uses apiKeyAuthentication Strict over exactly one key source in the release namespace (apiKeys.secretRef.name, .secretSelector.matchLabels or .configMapSelector.matchLabels, the last with hashed keys only). gateway.metricLabels.api_key (apiKey.name, a key entry's metadata.name) renders with it. The route's placement depends on the edge:

  • With the chart's own edge the route is on the HTTPS listener.
  • Behind a public Gateway it sits on a dedicated data-plane listener, llmRouting.external.listener (llm-external, 8082). The controller refuses a root model route on a listener with directly attached models. The public Gateway forwards the hostname there, and both policy flavours admit the port.

  The README says every key reaches every model until agentgateway enforces allowedModels on the Kubernetes API-key contract (giantswarm/giantswarm#37742).

• Guards and docs. Render guards: an empty model list, a duplicate name, a model without a baseURL, a wildcard the CRD refuses, the external endpoint with zero or two key sources, a hostPrefix that is no DNS label, and an external listener port taken by another listener. The schema refuses llmRouting.backend, .pathPrefixes and .routes. The connectivity README's LLM routing section, UPGRADE.md, CHANGELOG, both schemas and READMEs are updated.

What is the effect of this change to users?

An installation with llmRouting.enabled keeps routing claude-* models to Anthropic with the client's own key, at the same URL, and every request through the router is metered: agentgateway_gen_ai_client_token_usage and _cost_usd_total with the request and response model, the agent and the person. The series' route label becomes internal/llm:request. Served models join the listener by their preset name once model-manager#145 ships. An installation that set nothing renders nothing new. Breaking for an installation that set llmRouting.backend, pathPrefixes or routes: see UPGRADE.md.

How does it look like?

make verify-llm-routing covers:

• the listener (llm:8081:HTTP:Same:AgentgatewayModel), the models' parentRefs, and no in-cluster HTTPRoute or routes map;
• the default model and an installation's list, with the guards;
• the LLM endpoint document with and without external;
• the Role and the egress, including a node pool's topology (cluster-manager on, the slice off), and none of it without llmRouting or a serving model-manager;
• the external route at / in both edge shapes (with the dedicated listener, forwarder port and policy ports), the key policy with one source, and the api_key label.

make verify-all: 56 of 56.

Proven in a local agentlab (kind) with this chart and model-manager#145's build; the comment below has the output:

• kagent's default ModelConfig and an agent turn reach Anthropic through the router, and the turn's cost is recorded for the agent and the person.
• The CPU-served model answers Chat completions, Responses, Messages and count_tokens by its preset name.
• The external endpoint answers 401 without a key and with a wrong one, and 200 with the key; its series carry api_key.
• Every model is Accepted on both parents.

[circleci-architect]

Published Helm chart agent-platform-connectivity

4.70.2-rb83fe755t20260924220746h2eda23a

Chart	agent-platform-connectivity
Version	4.70.2-rb83fe755t20260924220746h2eda23a
OCI reference	oci://gsoci.azurecr.io/charts/giantswarm/agent-platform-connectivity:4.70.2-rb83fe755t20260924220746h2eda23a
Digest	sha256:7b2b98910053b2150841b9c6f3da8e20e796128f224220826ad993c27e54bbab
Registry	public — gsoci.azurecr.io
Git catalog	giantswarm-test-catalog (index)

Pull this chart

helm pull oci://gsoci.azurecr.io/charts/giantswarm/agent-platform-connectivity --version 4.70.2-rb83fe755t20260924220746h2eda23a

_{Posted by architect-orb · build 13547 · commit 2eda23a · updated in place on every push}

[circleci-architect]

Published Helm chart agent-platform

4.70.2-rb83fe755t20260924220746h2eda23a

Chart	agent-platform
Version	4.70.2-rb83fe755t20260924220746h2eda23a
OCI reference	oci://gsoci.azurecr.io/charts/giantswarm/agent-platform:4.70.2-rb83fe755t20260924220746h2eda23a
Digest	sha256:064cea11f827259e310a214e14328a9d69308ee2366e5f635e4b820ff7c1e879
Registry	public — gsoci.azurecr.io
Git catalog	giantswarm-test-catalog (index)

Pull this chart

helm pull oci://gsoci.azurecr.io/charts/giantswarm/agent-platform --version 4.70.2-rb83fe755t20260924220746h2eda23a

_{Posted by architect-orb · build 13548 · commit 2eda23a · updated in place on every push}

[teemow]

Lab proof: a fresh local agentlab (kind) with this branch's chart (b7fa12b), llmRouting and llmRouting.external on, and giantswarm/model-manager#173's build (dev-145-0f828c5).

Found and fixed in the lab (b7fa12b): the controller refused the listener-attached model with Accepted=False NotAllowedByListeners ("kind AgentgatewayModel is not allowed") until the LLM listener named the kind in allowedRoutes.kinds. After a reinstall from the committed chart, the listener's supported kinds are [agentgateway.dev AgentgatewayModel], and anthropic, qwen2-5-0-5b-cpu and qwen2-5-0-5b-cpu-workload are Accepted on both parents (the llm listener and agent-platform-connectivity-llm-external).

In-cluster listener (http://agentgateway.agent-platform.svc:8081, from a pod in the lab):

• POST /v1/messages claude-sonnet-4-6 with the client's key: 200 from Anthropic. POST /v1/chat/completions claude-sonnet-4-6: 200, converted to Anthropic.
• GET /v1/models: claude-* and qwen2-5-0-5b-cpu, not the Internal concrete model. An unknown model answers 404 model_not_found.
• By the served model's public name qwen2-5-0-5b-cpu, every answer is from Qwen/Qwen2.5-0.5B-Instruct: chat completions, Responses, Messages (stop_reason: end_turn, 35 in / 10 out) and count_tokens (input_tokens: 35).

External (https://llm.<domain>, the lab's own edge, route at /): GET /v1/models answers 401 without a key, 401 with a wrong key and 200 with the configured key. A chat completion by the public name with the key answers 200.

Metering. agentgateway_gen_ai_client_token_usage_count for input and output tokens:

request model	listener	agent	user	api_key
claude-sonnet-4-6	llm	kagent/default (the curl pod)	—	—
claude-sonnet-4-6	llm	kagent/agentlab-agents-test	admin@lab.local	—
Qwen/Qwen2.5-0.5B-Instruct	llm	kagent/default	—	—
Qwen/Qwen2.5-0.5B-Instruct	https	—	—	agent-one-proof

agentgateway_gen_ai_client_cost_usd_total for the agent's turn (agentlab agents-test, all PASS) was 0.0237 with agent="agentlab-agents-test", user="admin@lab.local". The series' route is internal/llm:request for the listener-attached models; the platform's dashboards select by cluster_id, gateway, agent and token type, not by route.