Skip to content

fix(serving): re-create a missing well-known LLMInferenceServiceConfig - #667

Merged
teemow merged 1 commit into
mainfrom
teemow/kserve-runtime-configs-drift
Sep 24, 2026
Merged

teemow merged 1 commit into
mainfrom
teemow/kserve-runtime-configs-drift

Conversation

@teemow

@teemow teemow commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

Towards #508

Problem

The kserve-runtime-configs HelmRelease carries no drift detection, so helm-controller judges it by its Helm storage alone. When the well-known LLMInferenceServiceConfigs disappear from the cluster (a slice torn down at once left them terminating under the llm-d controller's finalizer, the next install adopted them and the new controller finished their deletion), the release stays "in-sync" with no config behind it, and every LLMInferenceService composed from them fails PresetsCombined=False ConfigNotFound until a forced reconcile.

Change

  • components.kserve-runtime-configs.driftDetection.mode: enabled is the default in values.yaml, forwarded verbatim into the child HelmRelease's spec.driftDetection by templates/components.yaml. On every reconcile (gitops.interval) helm-controller re-applies a missing or changed config as a server-side apply, without a Helm revision.
  • No ignore rule: the configs are configuration only, the controller's finalizer survives a server-side apply of the manifest, and no webhook mutates them. The other serving releases keep helm-controller's default.
  • examples/serving-slice.yaml shows the setting; CHANGELOG entry; README and values.schema.json regenerated; the comments that named the set of drift-detecting releases name the third.
  • Tests: tests/verify-serving-slice.py asserts spec.driftDetection.mode: enabled on the slice's kserve-runtime-configs release and none on its other releases (a render without the default fails naming the release); tests/verify-kagent-wiring.py adds it to the set of releases that carry drift detection.

Proof

  • Local: make verify-serving-slice, make verify-meta, make verify-target pass; pre-commit on the changed files passes. A render without the default fails verify-serving-slice naming the release.
  • CI: green on 54e6999.
  • Live, a local kind lab with the serving slice on and this checkout's chart (CRs only, no GPU):
    • The kserve-runtime-configs HelmRelease carries spec.driftDetection: {mode: enabled}, and only kagent, muster and this release carry drift detection.
    • In sync: two requested reconciles log release in-sync with desired state. The configs carry serving.kserve.io/llmisvcconfig-finalizer, so the controller's finalizer is not reported as drift.
    • KServe's validating webhook refuses a direct delete (well-known config agent-platform/kserve-config-llm-tracing cannot be deleted). To delete one by hand, its DELETE operation was lifted and restored straight after the delete; the config was gone at 16:34:08Z.
    • On the release's next scheduled reconcile (10m interval, nothing requested, no --force) helm-controller logged detected changes in cluster state: removed: 1 and correct cluster drift, emitted DriftDetected and DriftCorrected, and re-created the config at 16:43:23Z (new uid, the controller's finalizer back on it). The webhook admitted the CREATE, and all ten configs were present.
    • Helm revision unchanged (v8). A following requested reconcile logs release in-sync with desired state with Drifted=False, so there is no perpetual drift. One benign Drifted condition patch conflict was logged and resolved on the immediate requeue.

components.kserve-runtime-configs carries driftDetection.mode: enabled by
default, so helm-controller re-applies a well-known config that went missing
on the release's next reconcile instead of reporting it in-sync.

Towards #508
@teemow
teemow requested a review from a team as a code owner September 24, 2026 15:47
@circleci-architect

Copy link
Copy Markdown

Published Helm chart agent-platform-connectivity

4.67.1-r9d846806t20260924154737h54e6999
Chart agent-platform-connectivity
Version 4.67.1-r9d846806t20260924154737h54e6999
OCI reference oci://gsoci.azurecr.io/charts/giantswarm/agent-platform-connectivity:4.67.1-r9d846806t20260924154737h54e6999
Digest sha256:faed62a452af02d2994a0a7e59509de50ed013a37dc44a2b064f584d253cde81
Registry public — gsoci.azurecr.io
Git catalog giantswarm-test-catalog (index)
Pull this chart
helm pull oci://gsoci.azurecr.io/charts/giantswarm/agent-platform-connectivity --version 4.67.1-r9d846806t20260924154737h54e6999

Posted by architect-orb · build 12777 · commit 54e6999 · updated in place on every push

@circleci-architect

Copy link
Copy Markdown

Published Helm chart agent-platform

4.67.1-r9d846806t20260924154737h54e6999
Chart agent-platform
Version 4.67.1-r9d846806t20260924154737h54e6999
OCI reference oci://gsoci.azurecr.io/charts/giantswarm/agent-platform:4.67.1-r9d846806t20260924154737h54e6999
Digest sha256:b2c2f06a819bf18a65db52b527d009cc2b7b9a4379ba7176c9dd252c4c05a975
Registry public — gsoci.azurecr.io
Git catalog giantswarm-test-catalog (index)
Pull this chart
helm pull oci://gsoci.azurecr.io/charts/giantswarm/agent-platform --version 4.67.1-r9d846806t20260924154737h54e6999

Posted by architect-orb · build 12774 · commit 54e6999 · updated in place on every push

@teemow
teemow merged commit 552d8a2 into main Sep 24, 2026
14 checks passed
@teemow
teemow deleted the teemow/kserve-runtime-configs-drift branch September 24, 2026 18:00
teemow added a commit that referenced this pull request Sep 24, 2026
…d-602

* origin/main:
  fix(model-serving): bound every emptyDir of a model pod with a sizeLimit (#670)
  fix(serving): re-create a missing well-known LLMInferenceServiceConfig (#667)

# Conflicts:
#	CHANGELOG.md
teemow added a commit that referenced this pull request Sep 24, 2026
…i-580

* origin/main:
  feat(model-serving): let model-manager read a served model's route list (#669)
  fix(model-serving): bound every emptyDir of a model pod with a sizeLimit (#670)
  fix(serving): re-create a missing well-known LLMInferenceServiceConfig (#667)
teemow added a commit that referenced this pull request Sep 24, 2026
…uter-603

* origin/main:
  feat(model-serving): let model-manager read a served model's route list (#669)
  fix(model-serving): bound every emptyDir of a model pod with a sizeLimit (#670)
  fix(serving): re-create a missing well-known LLMInferenceServiceConfig (#667)

# Conflicts:
#	CHANGELOG.md
teemow added a commit that referenced this pull request Sep 24, 2026
* origin/main:
  feat(model-serving): let model-manager read a served model's route list (#669)
  fix(model-serving): bound every emptyDir of a model pod with a sizeLimit (#670)
  fix(serving): re-create a missing well-known LLMInferenceServiceConfig (#667)

# Conflicts:
#	CHANGELOG.md
teemow added a commit that referenced this pull request Sep 24, 2026
* origin/main:
  feat(model-serving): let model-manager read a served model's route list (#669)
  fix(model-serving): bound every emptyDir of a model pod with a sizeLimit (#670)
  fix(serving): re-create a missing well-known LLMInferenceServiceConfig (#667)

# Conflicts:
#	CHANGELOG.md
#	Makefile.custom.mk
#	tests/verify-gpu-pool.py
teemow added a commit that referenced this pull request Sep 24, 2026
* origin/main:
  fix: drift detection ignores the serving configs' defaulted spec (#680)
  feat(model-serving): let model-manager read a served model's route list (#669)
  fix(model-serving): bound every emptyDir of a model pod with a sizeLimit (#670)
  fix(serving): re-create a missing well-known LLMInferenceServiceConfig (#667)

# Conflicts:
#	CHANGELOG.md
teemow added a commit that referenced this pull request Sep 24, 2026
…-oauth

* origin/main:
  fix(connectivity): limit the substrate-bootstrap hook to 256Mi (#666)
  fix: drift detection ignores the serving configs' defaulted spec (#680)
  feat(model-serving): let model-manager read a served model's route list (#669)
  fix(model-serving): bound every emptyDir of a model pod with a sizeLimit (#670)
  fix(serving): re-create a missing well-known LLMInferenceServiceConfig (#667)

# Conflicts:
#	CHANGELOG.md
teemow added a commit that referenced this pull request Sep 24, 2026
…ager-block-318

* origin/main:
  feat(observability): collect Substrate's and the kagent controller's metrics (#619)
  fix(connectivity): limit the substrate-bootstrap hook to 256Mi (#666)
  fix: drift detection ignores the serving configs' defaulted spec (#680)
  feat(model-serving): let model-manager read a served model's route list (#669)
  fix(model-serving): bound every emptyDir of a model pod with a sizeLimit (#670)
  fix(serving): re-create a missing well-known LLMInferenceServiceConfig (#667)

# Conflicts:
#	CHANGELOG.md
teemow added a commit that referenced this pull request Sep 24, 2026
…ardown-527

* origin/main:
  fix(agent-manager): cap the Generic agent chart below 1.5.0 (#668)
  feat(observability): collect Substrate's and the kagent controller's metrics (#619)
  fix(connectivity): limit the substrate-bootstrap hook to 256Mi (#666)
  fix: drift detection ignores the serving configs' defaulted spec (#680)
  feat(model-serving): let model-manager read a served model's route list (#669)
  fix(model-serving): bound every emptyDir of a model pod with a sizeLimit (#670)
  fix(serving): re-create a missing well-known LLMInferenceServiceConfig (#667)

# Conflicts:
#	CHANGELOG.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant