Skip to content

fix(connectivity): limit the substrate-bootstrap hook to 256Mi - #666

Merged
teemow merged 3 commits into
mainfrom
ap-513-hook-memory-limit
Sep 24, 2026
Merged

teemow merged 3 commits into
mainfrom
ap-513-hook-memory-limit

Conversation

@teemow

@teemow teemow commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

Towards giantswarm/agent-platform#513.

Problem

The connectivity chart's substrate-bootstrap hook Job ran with the hooks' shared container limit of 128Mi. Its script runs up to two kubectl processes at once (kubectl create --dry-run=client | kubectl apply --server-side for actor-id-ca-certs, kubectl into jq for the trust anchors), each some 30 MiB resident, next to jq, the shell and the key candidates on the in-memory /work, all charged to the pod's memory cgroup. On a busy node the Job's first pod was OOM-killed and the install completed only on the Job's retry; a hook that exhausts its backoffLimit fails the release.

Root cause

agent-platform.hooks.job (templates/_hooks.tpl) hard-coded one 128Mi limit for every hook's main container. That is enough for one kubectl run and its shell, and too small for the bootstrap's concurrent pipelines.

Proposed solution

  • agent-platform.hooks.job takes an optional memory: the main container's limit, default 128Mi.
  • templates/substrate/bootstrap.yaml passes 256Mi. Its header says why.
  • Every request stays 10m/32Mi. The openssl init container and the other three hooks (the derived Postgres Secrets, the model-serving cache claim, the pre-pull cleanup) keep 128Mi.
  • The hooks values comment lists all four hooks and their resources.
  • make verify-hooks-memory renders the four hook Jobs and asserts each container's requests and limits. It fails on main and passes here.
  • CHANGELOG entry under Unreleased / Fixed.
  • tests/verify-target.py compares the connectivity renders byte for byte with origin/main. It now holds this one intended difference: the bootstrap sh container's limit is blanked on both sides, in that Job only, until origin/main carries it.

Rendered before and after, over every ci/*.yaml of the connectivity chart with and without Substrate, plus a render with all four hooks on (56 renders): the only object that differs is t-substrate-bootstrap, and the only field that differs is its sh container's limits.memory (128Mi to 256Mi).

The script is not serialized. The limit is the smaller change and covers both pipelines.

This PR and #665 both change templates/substrate/bootstrap.yaml, in separate hunks. The two branches merge without conflict. #665's fourth CA pool adds one more sequential kubectl, not another concurrent one.

Acceptance criteria

  • helm template of the connectivity chart renders substrate-bootstrap with the raised limit and every other hook unchanged (make verify-hooks-memory, make verify-wiring).
  • A fresh local bring-up installs the chart and the substrate-bootstrap Job completes on its first pod.
  • The next connectivity release carries the change.

Checklist

  • Update changelog in CHANGELOG.md.
  • Make sure values.yaml and values.schema.json are valid (comment-only values change; the schema pre-commit hook passes).

The bootstrap script runs up to two kubectl processes at once next to
jq, the shell and its in-memory /work, all charged to the pod's memory
cgroup; at the hooks' shared 128Mi limit its first pod was OOM-killed
on a busy node and the install completed only on the Job's retry.

agent-platform.hooks.job takes an optional memory (the main container's
limit, default 128Mi); the bootstrap passes 256Mi. Requests and every
other hook stay unchanged. make verify-hooks-memory asserts the four
hook Jobs' resources.

Towards #513
@teemow
teemow requested a review from a team as a code owner September 24, 2026 15:43
verify-target compares the connectivity renders with origin/main byte for byte; the Substrate bootstrap's kubectl container limit is the one intended difference (#513), blanked on both sides in that Job only until origin/main carries it.
@circleci-architect

circleci-architect Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

Published Helm chart agent-platform

4.68.1-rd628c08ft20260924192649hebd6cb0
Chart agent-platform
Version 4.68.1-rd628c08ft20260924192649hebd6cb0
OCI reference oci://gsoci.azurecr.io/charts/giantswarm/agent-platform:4.68.1-rd628c08ft20260924192649hebd6cb0
Digest sha256:b9309cbbc56e98023a615805697c851677eceb691f2e09494fbd9d9b8b337c39
Registry public — gsoci.azurecr.io
Git catalog giantswarm-test-catalog (index)
Pull this chart
helm pull oci://gsoci.azurecr.io/charts/giantswarm/agent-platform --version 4.68.1-rd628c08ft20260924192649hebd6cb0

Posted by architect-orb · build 13120 · commit ebd6cb0 · updated in place on every push

@circleci-architect

circleci-architect Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

Published Helm chart agent-platform-connectivity

4.68.1-rd628c08ft20260924192649hebd6cb0
Chart agent-platform-connectivity
Version 4.68.1-rd628c08ft20260924192649hebd6cb0
OCI reference oci://gsoci.azurecr.io/charts/giantswarm/agent-platform-connectivity:4.68.1-rd628c08ft20260924192649hebd6cb0
Digest sha256:12eb33f72dc669a679522830e56066d85538c3468ad47987828250ee66f52014
Registry public — gsoci.azurecr.io
Git catalog giantswarm-test-catalog (index)
Pull this chart
helm pull oci://gsoci.azurecr.io/charts/giantswarm/agent-platform-connectivity --version 4.68.1-rd628c08ft20260924192649hebd6cb0

Posted by architect-orb · build 13122 · commit ebd6cb0 · updated in place on every push

@teemow

teemow commented Sep 24, 2026

Copy link
Copy Markdown
Member Author

Lab proof for the second acceptance criterion, head f32e088.

A fresh agentlab bring-up (agentlab v0.57.9, new kind cluster) with this branch's dev build: meta chart 4.67.1-rd628c08ft20260924154924hf32e088 and its connectivity chart at the same version. Agents on, no values overlay.

  • agentlab up: exit 0. All 12 HelmReleases Ready, every Helm release deployed. agent-platform-connectivity is at revision 1.
  • Pods, Jobs and events were watched from before the platform install until the end of up, because hook-succeeded deletes the Job.
  • agent-platform-connectivity-substrate-bootstrap completed on its first pod. There was one Job and one pod. Its events are one SuccessfulCreate, then Completed. Status: succeeded 1, failed 0. It ran from 16:28:27Z to 16:28:31Z UTC. The openssl init container and the sh container both exited 0 with no restarts. No second pod was created.
  • Rendered limits (helm get hooks agent-platform-connectivity, same as the pod spec): the bootstrap sh container requests 10m/32Mi with a 256Mi limit. The bootstrap openssl init container and the postgres-databases hook stay at 10m/32Mi with 128Mi.
  • Kind node dmesg: the ring buffer runs from boot and has not wrapped. No line matches oom-kill|out of memory|killed process|invoked oom.

The lab was deleted after the run.

* origin/main:
  feat(model-serving): let model-manager read a served model's route list (#669)
  fix(model-serving): bound every emptyDir of a model pod with a sizeLimit (#670)
  fix(serving): re-create a missing well-known LLMInferenceServiceConfig (#667)

# Conflicts:
#	CHANGELOG.md
@teemow
teemow merged commit 3c69c86 into main Sep 24, 2026
13 checks passed
@teemow
teemow deleted the ap-513-hook-memory-limit branch September 24, 2026 20:07
teemow added a commit that referenced this pull request Sep 24, 2026
…-oauth

* origin/main:
  fix(connectivity): limit the substrate-bootstrap hook to 256Mi (#666)
  fix: drift detection ignores the serving configs' defaulted spec (#680)
  feat(model-serving): let model-manager read a served model's route list (#669)
  fix(model-serving): bound every emptyDir of a model pod with a sizeLimit (#670)
  fix(serving): re-create a missing well-known LLMInferenceServiceConfig (#667)

# Conflicts:
#	CHANGELOG.md
teemow added a commit that referenced this pull request Sep 24, 2026
…ager-block-318

* origin/main:
  feat(observability): collect Substrate's and the kagent controller's metrics (#619)
  fix(connectivity): limit the substrate-bootstrap hook to 256Mi (#666)
  fix: drift detection ignores the serving configs' defaulted spec (#680)
  feat(model-serving): let model-manager read a served model's route list (#669)
  fix(model-serving): bound every emptyDir of a model pod with a sizeLimit (#670)
  fix(serving): re-create a missing well-known LLMInferenceServiceConfig (#667)

# Conflicts:
#	CHANGELOG.md
teemow added a commit that referenced this pull request Sep 24, 2026
…ardown-527

* origin/main:
  fix(agent-manager): cap the Generic agent chart below 1.5.0 (#668)
  feat(observability): collect Substrate's and the kagent controller's metrics (#619)
  fix(connectivity): limit the substrate-bootstrap hook to 256Mi (#666)
  fix: drift detection ignores the serving configs' defaulted spec (#680)
  feat(model-serving): let model-manager read a served model's route list (#669)
  fix(model-serving): bound every emptyDir of a model pod with a sizeLimit (#670)
  fix(serving): re-create a missing well-known LLMInferenceServiceConfig (#667)

# Conflicts:
#	CHANGELOG.md
teemow added a commit that referenced this pull request Sep 24, 2026
* origin/main:
  fix(agent-manager): cap the Generic agent chart below 1.5.0 (#668)
  feat(observability): collect Substrate's and the kagent controller's metrics (#619)
  fix(connectivity): limit the substrate-bootstrap hook to 256Mi (#666)
  fix: drift detection ignores the serving configs' defaulted spec (#680)

# Conflicts:
#	CHANGELOG.md
teemow added a commit that referenced this pull request Sep 24, 2026
…follows-kserve-602

* origin/main:
  fix(backstage): drop the portal's dead agentPlatform.modelManager block (#677)
  fix(agent-manager): cap the Generic agent chart below 1.5.0 (#668)
  feat(observability): collect Substrate's and the kagent controller's metrics (#619)
  fix(connectivity): limit the substrate-bootstrap hook to 256Mi (#666)
  fix: drift detection ignores the serving configs' defaulted spec (#680)

# Conflicts:
#	CHANGELOG.md
teemow added a commit that referenced this pull request Sep 24, 2026
…uter-603

* origin/main:
  fix(model-manager): read served models wherever model-manager serves (#681)
  feat(model-serving): check every shipped preset's parsers by model family (#674)
  fix(backstage): drop the portal's dead agentPlatform.modelManager block (#677)
  fix(agent-manager): cap the Generic agent chart below 1.5.0 (#668)
  feat(observability): collect Substrate's and the kagent controller's metrics (#619)
  fix(connectivity): limit the substrate-bootstrap hook to 256Mi (#666)
  fix: drift detection ignores the serving configs' defaulted spec (#680)

# Conflicts:
#	CHANGELOG.md
#	UPGRADE.md
teemow added a commit that referenced this pull request Sep 24, 2026
* origin/main:
  feat: the kagent line and the Substrate line at 1.1.0 (#665)
  fix(model-manager): read served models wherever model-manager serves (#681)
  feat(model-serving): check every shipped preset's parsers by model family (#674)
  fix(backstage): drop the portal's dead agentPlatform.modelManager block (#677)
  fix(agent-manager): cap the Generic agent chart below 1.5.0 (#668)
  feat(observability): collect Substrate's and the kagent controller's metrics (#619)
  fix(connectivity): limit the substrate-bootstrap hook to 256Mi (#666)

# Conflicts:
#	CHANGELOG.md
teemow added a commit that referenced this pull request Sep 25, 2026
…i-580

* origin/main:
  fix(serving): tear the serving slice down in order, configs freed (#679)
  fix: derive the managers' OAuth login from muster's OAuth server (#678)
  feat(model-serving): stamp the cache claim's tier as annotations (#676)
  fix(substrate): the capz snapshot-store identity holds Storage Blob Delegator on the account (#686)
  fix(connectivity): the egress gateway reaches the Substrate credential provider (#685)
  feat(llm-routing): route the LLM endpoint by model, with API keys (#672)
  feat: the kagent line and the Substrate line at 1.1.0 (#665)
  fix(model-manager): read served models wherever model-manager serves (#681)
  feat(model-serving): check every shipped preset's parsers by model family (#674)
  fix(backstage): drop the portal's dead agentPlatform.modelManager block (#677)
  fix(agent-manager): cap the Generic agent chart below 1.5.0 (#668)
  feat(observability): collect Substrate's and the kagent controller's metrics (#619)
  fix(connectivity): limit the substrate-bootstrap hook to 256Mi (#666)
  fix: drift detection ignores the serving configs' defaulted spec (#680)

# Conflicts:
#	CHANGELOG.md
#	tests/verify-target.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant