Repository navigation
fix(connectivity): limit the substrate-bootstrap hook to 256Mi - #666
Merged
Merged
Conversation
The bootstrap script runs up to two kubectl processes at once next to jq, the shell and its in-memory /work, all charged to the pod's memory cgroup; at the hooks' shared 128Mi limit its first pod was OOM-killed on a busy node and the install completed only on the Job's retry. agent-platform.hooks.job takes an optional memory (the main container's limit, default 128Mi); the bootstrap passes 256Mi. Requests and every other hook stay unchanged. make verify-hooks-memory asserts the four hook Jobs' resources. Towards #513
verify-target compares the connectivity renders with origin/main byte for byte; the Substrate bootstrap's kubectl container limit is the one intended difference (#513), blanked on both sides in that Job only until origin/main carries it.
Published Helm chart
|
| Chart | agent-platform |
| Version | 4.68.1-rd628c08ft20260924192649hebd6cb0 |
| OCI reference | oci://gsoci.azurecr.io/charts/giantswarm/agent-platform:4.68.1-rd628c08ft20260924192649hebd6cb0 |
| Digest | sha256:b9309cbbc56e98023a615805697c851677eceb691f2e09494fbd9d9b8b337c39 |
| Registry | public — gsoci.azurecr.io |
| Git catalog | giantswarm-test-catalog (index) |
Pull this chart
helm pull oci://gsoci.azurecr.io/charts/giantswarm/agent-platform --version 4.68.1-rd628c08ft20260924192649hebd6cb0Posted by architect-orb · build 13120 · commit ebd6cb0 · updated in place on every push
Published Helm chart
|
| Chart | agent-platform-connectivity |
| Version | 4.68.1-rd628c08ft20260924192649hebd6cb0 |
| OCI reference | oci://gsoci.azurecr.io/charts/giantswarm/agent-platform-connectivity:4.68.1-rd628c08ft20260924192649hebd6cb0 |
| Digest | sha256:12eb33f72dc669a679522830e56066d85538c3468ad47987828250ee66f52014 |
| Registry | public — gsoci.azurecr.io |
| Git catalog | giantswarm-test-catalog (index) |
Pull this chart
helm pull oci://gsoci.azurecr.io/charts/giantswarm/agent-platform-connectivity --version 4.68.1-rd628c08ft20260924192649hebd6cb0Posted by architect-orb · build 13122 · commit ebd6cb0 · updated in place on every push
Member
Author
|
Lab proof for the second acceptance criterion, head A fresh agentlab bring-up (agentlab v0.57.9, new kind cluster) with this branch's dev build: meta chart
The lab was deleted after the run. |
teemow
added a commit
that referenced
this pull request
Sep 24, 2026
…-oauth * origin/main: fix(connectivity): limit the substrate-bootstrap hook to 256Mi (#666) fix: drift detection ignores the serving configs' defaulted spec (#680) feat(model-serving): let model-manager read a served model's route list (#669) fix(model-serving): bound every emptyDir of a model pod with a sizeLimit (#670) fix(serving): re-create a missing well-known LLMInferenceServiceConfig (#667) # Conflicts: # CHANGELOG.md
3 tasks done
teemow
added a commit
that referenced
this pull request
Sep 24, 2026
…ager-block-318 * origin/main: feat(observability): collect Substrate's and the kagent controller's metrics (#619) fix(connectivity): limit the substrate-bootstrap hook to 256Mi (#666) fix: drift detection ignores the serving configs' defaulted spec (#680) feat(model-serving): let model-manager read a served model's route list (#669) fix(model-serving): bound every emptyDir of a model pod with a sizeLimit (#670) fix(serving): re-create a missing well-known LLMInferenceServiceConfig (#667) # Conflicts: # CHANGELOG.md
teemow
added a commit
that referenced
this pull request
Sep 24, 2026
…ardown-527 * origin/main: fix(agent-manager): cap the Generic agent chart below 1.5.0 (#668) feat(observability): collect Substrate's and the kagent controller's metrics (#619) fix(connectivity): limit the substrate-bootstrap hook to 256Mi (#666) fix: drift detection ignores the serving configs' defaulted spec (#680) feat(model-serving): let model-manager read a served model's route list (#669) fix(model-serving): bound every emptyDir of a model pod with a sizeLimit (#670) fix(serving): re-create a missing well-known LLMInferenceServiceConfig (#667) # Conflicts: # CHANGELOG.md
teemow
added a commit
that referenced
this pull request
Sep 24, 2026
* origin/main: fix(agent-manager): cap the Generic agent chart below 1.5.0 (#668) feat(observability): collect Substrate's and the kagent controller's metrics (#619) fix(connectivity): limit the substrate-bootstrap hook to 256Mi (#666) fix: drift detection ignores the serving configs' defaulted spec (#680) # Conflicts: # CHANGELOG.md
teemow
added a commit
that referenced
this pull request
Sep 24, 2026
…follows-kserve-602 * origin/main: fix(backstage): drop the portal's dead agentPlatform.modelManager block (#677) fix(agent-manager): cap the Generic agent chart below 1.5.0 (#668) feat(observability): collect Substrate's and the kagent controller's metrics (#619) fix(connectivity): limit the substrate-bootstrap hook to 256Mi (#666) fix: drift detection ignores the serving configs' defaulted spec (#680) # Conflicts: # CHANGELOG.md
teemow
added a commit
that referenced
this pull request
Sep 24, 2026
…uter-603 * origin/main: fix(model-manager): read served models wherever model-manager serves (#681) feat(model-serving): check every shipped preset's parsers by model family (#674) fix(backstage): drop the portal's dead agentPlatform.modelManager block (#677) fix(agent-manager): cap the Generic agent chart below 1.5.0 (#668) feat(observability): collect Substrate's and the kagent controller's metrics (#619) fix(connectivity): limit the substrate-bootstrap hook to 256Mi (#666) fix: drift detection ignores the serving configs' defaulted spec (#680) # Conflicts: # CHANGELOG.md # UPGRADE.md
teemow
added a commit
that referenced
this pull request
Sep 24, 2026
* origin/main: feat: the kagent line and the Substrate line at 1.1.0 (#665) fix(model-manager): read served models wherever model-manager serves (#681) feat(model-serving): check every shipped preset's parsers by model family (#674) fix(backstage): drop the portal's dead agentPlatform.modelManager block (#677) fix(agent-manager): cap the Generic agent chart below 1.5.0 (#668) feat(observability): collect Substrate's and the kagent controller's metrics (#619) fix(connectivity): limit the substrate-bootstrap hook to 256Mi (#666) # Conflicts: # CHANGELOG.md
teemow
added a commit
that referenced
this pull request
Sep 25, 2026
…i-580 * origin/main: fix(serving): tear the serving slice down in order, configs freed (#679) fix: derive the managers' OAuth login from muster's OAuth server (#678) feat(model-serving): stamp the cache claim's tier as annotations (#676) fix(substrate): the capz snapshot-store identity holds Storage Blob Delegator on the account (#686) fix(connectivity): the egress gateway reaches the Substrate credential provider (#685) feat(llm-routing): route the LLM endpoint by model, with API keys (#672) feat: the kagent line and the Substrate line at 1.1.0 (#665) fix(model-manager): read served models wherever model-manager serves (#681) feat(model-serving): check every shipped preset's parsers by model family (#674) fix(backstage): drop the portal's dead agentPlatform.modelManager block (#677) fix(agent-manager): cap the Generic agent chart below 1.5.0 (#668) feat(observability): collect Substrate's and the kagent controller's metrics (#619) fix(connectivity): limit the substrate-bootstrap hook to 256Mi (#666) fix: drift detection ignores the serving configs' defaulted spec (#680) # Conflicts: # CHANGELOG.md # tests/verify-target.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Towards giantswarm/agent-platform#513.
Problem
The connectivity chart's
substrate-bootstraphook Job ran with the hooks' shared container limit of128Mi. Its script runs up to two kubectl processes at once (kubectl create --dry-run=client | kubectl apply --server-sideforactor-id-ca-certs, kubectl into jq for the trust anchors), each some 30 MiB resident, next to jq, the shell and the key candidates on the in-memory/work, all charged to the pod's memory cgroup. On a busy node the Job's first pod was OOM-killed and the install completed only on the Job's retry; a hook that exhausts itsbackoffLimitfails the release.Root cause
agent-platform.hooks.job(templates/_hooks.tpl) hard-coded one128Milimit for every hook's main container. That is enough for one kubectl run and its shell, and too small for the bootstrap's concurrent pipelines.Proposed solution
agent-platform.hooks.jobtakes an optionalmemory: the main container's limit, default128Mi.templates/substrate/bootstrap.yamlpasses256Mi. Its header says why.10m/32Mi. The openssl init container and the other three hooks (the derived Postgres Secrets, the model-serving cache claim, the pre-pull cleanup) keep128Mi.hooksvalues comment lists all four hooks and their resources.make verify-hooks-memoryrenders the four hook Jobs and asserts each container's requests and limits. It fails onmainand passes here.tests/verify-target.pycompares the connectivity renders byte for byte withorigin/main. It now holds this one intended difference: the bootstrapshcontainer's limit is blanked on both sides, in that Job only, untilorigin/maincarries it.Rendered before and after, over every
ci/*.yamlof the connectivity chart with and without Substrate, plus a render with all four hooks on (56 renders): the only object that differs ist-substrate-bootstrap, and the only field that differs is itsshcontainer'slimits.memory(128Mito256Mi).The script is not serialized. The limit is the smaller change and covers both pipelines.
This PR and #665 both change
templates/substrate/bootstrap.yaml, in separate hunks. The two branches merge without conflict. #665's fourth CA pool adds one more sequential kubectl, not another concurrent one.Acceptance criteria
helm templateof the connectivity chart renderssubstrate-bootstrapwith the raised limit and every other hook unchanged (make verify-hooks-memory,make verify-wiring).substrate-bootstrapJob completes on its first pod.Checklist
values.yamlandvalues.schema.jsonare valid (comment-only values change; the schema pre-commit hook passes).