Repository navigation
fix(serving): re-create a missing well-known LLMInferenceServiceConfig - #667
Merged
Merged
Conversation
components.kserve-runtime-configs carries driftDetection.mode: enabled by default, so helm-controller re-applies a well-known config that went missing on the release's next reconcile instead of reporting it in-sync. Towards #508
Published Helm chart
|
| Chart | agent-platform-connectivity |
| Version | 4.67.1-r9d846806t20260924154737h54e6999 |
| OCI reference | oci://gsoci.azurecr.io/charts/giantswarm/agent-platform-connectivity:4.67.1-r9d846806t20260924154737h54e6999 |
| Digest | sha256:faed62a452af02d2994a0a7e59509de50ed013a37dc44a2b064f584d253cde81 |
| Registry | public — gsoci.azurecr.io |
| Git catalog | giantswarm-test-catalog (index) |
Pull this chart
helm pull oci://gsoci.azurecr.io/charts/giantswarm/agent-platform-connectivity --version 4.67.1-r9d846806t20260924154737h54e6999Posted by architect-orb · build 12777 · commit 54e6999 · updated in place on every push
Published Helm chart
|
| Chart | agent-platform |
| Version | 4.67.1-r9d846806t20260924154737h54e6999 |
| OCI reference | oci://gsoci.azurecr.io/charts/giantswarm/agent-platform:4.67.1-r9d846806t20260924154737h54e6999 |
| Digest | sha256:b2c2f06a819bf18a65db52b527d009cc2b7b9a4379ba7176c9dd252c4c05a975 |
| Registry | public — gsoci.azurecr.io |
| Git catalog | giantswarm-test-catalog (index) |
Pull this chart
helm pull oci://gsoci.azurecr.io/charts/giantswarm/agent-platform --version 4.67.1-r9d846806t20260924154737h54e6999Posted by architect-orb · build 12774 · commit 54e6999 · updated in place on every push
teemow
added a commit
that referenced
this pull request
Sep 24, 2026
* origin/main: feat(model-serving): let model-manager read a served model's route list (#669) fix(model-serving): bound every emptyDir of a model pod with a sizeLimit (#670) fix(serving): re-create a missing well-known LLMInferenceServiceConfig (#667) # Conflicts: # CHANGELOG.md # Makefile.custom.mk # tests/verify-gpu-pool.py
teemow
added a commit
that referenced
this pull request
Sep 24, 2026
* origin/main: fix: drift detection ignores the serving configs' defaulted spec (#680) feat(model-serving): let model-manager read a served model's route list (#669) fix(model-serving): bound every emptyDir of a model pod with a sizeLimit (#670) fix(serving): re-create a missing well-known LLMInferenceServiceConfig (#667) # Conflicts: # CHANGELOG.md
teemow
added a commit
that referenced
this pull request
Sep 24, 2026
…-oauth * origin/main: fix(connectivity): limit the substrate-bootstrap hook to 256Mi (#666) fix: drift detection ignores the serving configs' defaulted spec (#680) feat(model-serving): let model-manager read a served model's route list (#669) fix(model-serving): bound every emptyDir of a model pod with a sizeLimit (#670) fix(serving): re-create a missing well-known LLMInferenceServiceConfig (#667) # Conflicts: # CHANGELOG.md
teemow
added a commit
that referenced
this pull request
Sep 24, 2026
…ager-block-318 * origin/main: feat(observability): collect Substrate's and the kagent controller's metrics (#619) fix(connectivity): limit the substrate-bootstrap hook to 256Mi (#666) fix: drift detection ignores the serving configs' defaulted spec (#680) feat(model-serving): let model-manager read a served model's route list (#669) fix(model-serving): bound every emptyDir of a model pod with a sizeLimit (#670) fix(serving): re-create a missing well-known LLMInferenceServiceConfig (#667) # Conflicts: # CHANGELOG.md
teemow
added a commit
that referenced
this pull request
Sep 24, 2026
…ardown-527 * origin/main: fix(agent-manager): cap the Generic agent chart below 1.5.0 (#668) feat(observability): collect Substrate's and the kagent controller's metrics (#619) fix(connectivity): limit the substrate-bootstrap hook to 256Mi (#666) fix: drift detection ignores the serving configs' defaulted spec (#680) feat(model-serving): let model-manager read a served model's route list (#669) fix(model-serving): bound every emptyDir of a model pod with a sizeLimit (#670) fix(serving): re-create a missing well-known LLMInferenceServiceConfig (#667) # Conflicts: # CHANGELOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Towards #508
Problem
The
kserve-runtime-configsHelmRelease carries no drift detection, so helm-controller judges it by its Helm storage alone. When the well-knownLLMInferenceServiceConfigs disappear from the cluster (a slice torn down at once left them terminating under the llm-d controller's finalizer, the next install adopted them and the new controller finished their deletion), the release stays "in-sync" with no config behind it, and everyLLMInferenceServicecomposed from them failsPresetsCombined=False ConfigNotFounduntil a forced reconcile.Change
components.kserve-runtime-configs.driftDetection.mode: enabledis the default invalues.yaml, forwarded verbatim into the child HelmRelease'sspec.driftDetectionbytemplates/components.yaml. On every reconcile (gitops.interval) helm-controller re-applies a missing or changed config as a server-side apply, without a Helm revision.ignorerule: the configs are configuration only, the controller's finalizer survives a server-side apply of the manifest, and no webhook mutates them. The other serving releases keep helm-controller's default.examples/serving-slice.yamlshows the setting; CHANGELOG entry; README andvalues.schema.jsonregenerated; the comments that named the set of drift-detecting releases name the third.tests/verify-serving-slice.pyassertsspec.driftDetection.mode: enabledon the slice'skserve-runtime-configsrelease and none on its other releases (a render without the default fails naming the release);tests/verify-kagent-wiring.pyadds it to the set of releases that carry drift detection.Proof
make verify-serving-slice,make verify-meta,make verify-targetpass;pre-commiton the changed files passes. A render without the default failsverify-serving-slicenaming the release.54e6999.kserve-runtime-configsHelmRelease carriesspec.driftDetection: {mode: enabled}, and only kagent, muster and this release carry drift detection.release in-sync with desired state. The configs carryserving.kserve.io/llmisvcconfig-finalizer, so the controller's finalizer is not reported as drift.well-known config agent-platform/kserve-config-llm-tracing cannot be deleted). To delete one by hand, its DELETE operation was lifted and restored straight after the delete; the config was gone at 16:34:08Z.--force) helm-controller loggeddetected changes in cluster state: removed: 1andcorrect cluster drift, emittedDriftDetectedandDriftCorrected, and re-created the config at 16:43:23Z (new uid, the controller's finalizer back on it). The webhook admitted the CREATE, and all ten configs were present.release in-sync with desired statewithDrifted=False, so there is no perpetual drift. One benignDriftedcondition patch conflict was logged and resolved on the immediate requeue.