Skip to content

fix(hooks): serving teardown retries once Flux's API is gone - #707

Merged
teemow merged 1 commit into
mainfrom
teemow/serving-teardown-retry
Sep 26, 2026
Merged

teemow merged 1 commit into
mainfrom
teemow/serving-teardown-retry

Conversation

@teemow

@teemow teemow commented Sep 26, 2026

Copy link
Copy Markdown
Member

Towards #527

Problem

The <release>-serving-teardown hook (pre-delete) deletes the kserve-llmisvc-resources and kserve-runtime-configs HelmReleases unconditionally. With the bundled engine on, an uninstall that fails after the engine's teardown (it removes Flux, CRDs included) is retried with the hooks running again, and every attempt of the serving hook then fails:

1/3 the llm-d controller's release kserve-llmisvc-resources: its webhook denies every delete of a well-known config while it runs
error: the server doesn't have a resource type "helmreleases" in group "helm.toolkit.fluxcd.io"

So a failed uninstall cannot be retried: the retry stops at this hook (Job Failed. failed: 3/1).

Change

The two child release deletes run only while Flux's HelmRelease CRD exists. Without it no child release is left to order, and the hook says so. The config step is unchanged: it still frees any config left with the llm-d finalizer. make verify-serving-teardown adds the retry: with the HelmRelease API gone the script completes and deletes nothing.

Proof

  • make verify-serving-teardown: the order, the storage-version requests, the idempotent re-run, and the new retry without the HelmRelease API.
  • Local kind lab (agent-platform 4.76.1, bundled engine, serving slice on, the llm-d controller running, ten configs): helm uninstall agent-platform. The serving hook ran in 8 s in cluster-manager's order: the controller's release deleted and its webhook gone, the ten configs deleted through v1alpha2 with the finalizer removed, then kserve-runtime-configs. Afterwards there was no kserve-* HelmRelease, no LLMInferenceServiceConfig, no llmisvc webhook configuration and no kserve-* release Secret. The retry of the same uninstall then failed at this hook as shown above; that is what this PR fixes.

The meta release's own uninstall failing in the bundled-engine shape (Helm can't build the release's HelmRelease and OCIRepository objects once Flux's CRDs are gone) is a separate defect, filed on its own.

@circleci-architect

Copy link
Copy Markdown

Published Helm chart agent-platform

4.80.1-r2e419057t20260926071816hb919a06
Chart agent-platform
Version 4.80.1-r2e419057t20260926071816hb919a06
OCI reference oci://gsoci.azurecr.io/charts/giantswarm/agent-platform:4.80.1-r2e419057t20260926071816hb919a06
Digest sha256:567743d822b892dbd2c73976f9ee15302bd86ecbc03b5f3f733e4f4e5ae47f58
Registry public — gsoci.azurecr.io
Git catalog giantswarm-test-catalog (index)
Pull this chart
helm pull oci://gsoci.azurecr.io/charts/giantswarm/agent-platform --version 4.80.1-r2e419057t20260926071816hb919a06

Posted by architect-orb · build 14637 · commit b919a06 · updated in place on every push

@circleci-architect

Copy link
Copy Markdown

Published Helm chart agent-platform-connectivity

4.80.1-r2e419057t20260926071816hb919a06
Chart agent-platform-connectivity
Version 4.80.1-r2e419057t20260926071816hb919a06
OCI reference oci://gsoci.azurecr.io/charts/giantswarm/agent-platform-connectivity:4.80.1-r2e419057t20260926071816hb919a06
Digest sha256:dc16bf38ed38172bdc80d26e29e6e640af0e1df08deb4b273fc77ba55834dc8c
Registry public — gsoci.azurecr.io
Git catalog giantswarm-test-catalog (index)
Pull this chart
helm pull oci://gsoci.azurecr.io/charts/giantswarm/agent-platform-connectivity --version 4.80.1-r2e419057t20260926071816hb919a06

Posted by architect-orb · build 14638 · commit b919a06 · updated in place on every push

@teemow
teemow merged commit e266c43 into main Sep 26, 2026
13 checks passed
@teemow
teemow deleted the teemow/serving-teardown-retry branch September 26, 2026 13:55
teemow added a commit that referenced this pull request Sep 26, 2026
…en-606

* origin/main:
  fix(hooks): serving teardown retries once Flux's API is gone (#707)
  feat(cluster-manager): provider aws opens the CAPA API server LBs (#704)
  feat: the kagent line at 1.2.0 with its SDK-spec OTel values (#693)

# Conflicts:
#	CHANGELOG.md
teemow added a commit that referenced this pull request Sep 26, 2026
…reset-597

* origin/main:
  fix(model-serving): gpt-oss reads its tiktoken encodings from the image (#675)
  fix(hooks): serving teardown retries once Flux's API is gone (#707)
  feat(cluster-manager): provider aws opens the CAPA API server LBs (#704)
  feat: the kagent line at 1.2.0 with its SDK-spec OTel values (#693)

# Conflicts:
#	CHANGELOG.md
#	tests/verify-gpu-pool.py
teemow added a commit that referenced this pull request Sep 26, 2026
…se-placement

* origin/main:
  fix(model-serving): gpt-oss reads its tiktoken encodings from the image (#675)
  fix(hooks): serving teardown retries once Flux's API is gone (#707)
  feat(cluster-manager): provider aws opens the CAPA API server LBs (#704)
teemow added a commit that referenced this pull request Sep 26, 2026
…8-kv-611

* origin/main:
  feat(model-serving): the muse-glimmer-30b preset on vLLM's own image (#673)
  fix(model-serving): gpt-oss reads its tiktoken encodings from the image (#675)
  fix(hooks): serving teardown retries once Flux's API is gone (#707)
  feat(cluster-manager): provider aws opens the CAPA API server LBs (#704)
  feat: the kagent line at 1.2.0 with its SDK-spec OTel values (#693)

# Conflicts:
#	README.md
#	tests/verify-gpu-pool.py
teemow added a commit that referenced this pull request Sep 26, 2026
…okens

* origin/main:
  feat(connectivity): retire the model-manager REST route (#711)
  feat(model-serving): gemma-4-31b at 32k with an fp8 KV cache (#709)
  feat(model-serving): the muse-glimmer-30b preset on vLLM's own image (#673)
  fix(model-serving): gpt-oss reads its tiktoken encodings from the image (#675)
  fix(hooks): serving teardown retries once Flux's API is gone (#707)
  feat(cluster-manager): provider aws opens the CAPA API server LBs (#704)

# Conflicts:
#	CHANGELOG.md
teemow added a commit that referenced this pull request Sep 26, 2026
…n-post-delete

* origin/main:
  fix(connectivity): catalog ModelConfigs inherit the platform maxTokens (#712)
  feat(connectivity): retire the model-manager REST route (#711)
  feat(model-serving): gemma-4-31b at 32k with an fp8 KV cache (#709)
  feat(model-serving): the muse-glimmer-30b preset on vLLM's own image (#673)
  fix(model-serving): gpt-oss reads its tiktoken encodings from the image (#675)
  fix(hooks): serving teardown retries once Flux's API is gone (#707)
  feat(cluster-manager): provider aws opens the CAPA API server LBs (#704)

Signed-off-by: Timo Derstappen <teemow@gmail.com>

# Conflicts:
#	CHANGELOG.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant