Skip to content

fix(meta): the meta chart's own hook Jobs have no network policy — under Cilium policyEnforcementMode always every upgrade since 4.7.13 fails on the kagent-storage-version-backup hook #413

Description

@teemow

Problem

Since 4.7.13 (#398) the meta chart renders hook Jobs of its own that a cluster's own Flux runs too: <release>-kagent-storage-version-backup (pre-install,pre-upgrade, weight -7) and <release>-kagent-storage-version-restore (post-install,post-upgrade), whenever components.kagent-crds is on. With the bundled engine there are more (kagent-namespace, the self hooks, the teardown). All of them run kubectl against the apiserver from the release namespace under the pod labels app.kubernetes.io/instance: <release> + app.kubernetes.io/component: hooks, and the chart renders no network policy for that identity — networkPolicy.* governs the connectivity chart's policies only; the meta chart's templates/ carry none.

On a cluster whose Cilium runs policyEnforcementMode: always the hook pod is default-denied, kubectl loops on dial tcp 10.96.0.1:443: i/o timeout, the Job hits its activeDeadlineSeconds, and the upgrade fails with pre-upgrade hooks failed: timeout waiting for: [Job/<ns>/<release>-kagent-storage-version-backup status: 'InProgress']. The first 4.x installation has been failing every meta-chart upgrade since v4.7.13 (2026-09-11 23:55Z): last deployed revision 4.7.12, ~130 failed revisions through 4.9.0. Its component releases kept auto-upgrading through their own version ranges, so nothing looked broken until the meta release's status was read — but none of the meta chart's own changes since 4.7.13 (the kagent values hand-over of #407/#408, the drift detection of #410, atelet's pinned image set of #405, the store block of #412) reached the cluster.

This is the meta-chart sibling of #367 (the connectivity chart's substrate-bootstrap / postgres-databases hooks, same gap, same shape). The installation-side workaround there (a hand-written policy on app.kubernetes.io/instance: <connectivity release>) does not select the meta release's hooks, which carry the meta release's instance label.

Proposed solution

Render a network policy for the hook identity as a hook object — a pre-install/pre-upgrade hook runs before the release's manifests are applied, so a regular chart object could never cover the first run: helm.sh/hook on the same events as the hook ServiceAccount (agent-platform.hooks.serviceAccountEvents), weight -10 next to the ServiceAccount + ClusterRoleBinding in templates/hooks/rbac.yaml, delete policy before-hook-creation,hook-succeeded. Selector app.kubernetes.io/instance: <release> + app.kubernetes.io/component: hooks; egress to the apiserver only (toEntities: [kube-apiserver] in the cilium flavour, the apiserver endpoints in the kubernetes flavour), honouring networkPolicy.enabled / networkPolicy.flavor (auto resolves as it does for the connectivity chart). Rendered exactly when rbac.yaml renders the identity — the two describe the same pods.

Acceptance criteria

  • helm template of the default values renders the hook network policy in the cilium flavour (--api-versions cilium.io/v2) and in the kubernetes flavour, as a hook object with weight -10, the hook identity's events and delete policy; nothing renders with networkPolicy.enabled: false or when no hook renders.
  • The policy selects exactly the hook identity's pods (app.kubernetes.io/instance: <release>, app.kubernetes.io/component: hooks) and admits egress to the apiserver only.
  • A verify target asserts the policy's presence and shape in both flavours and its absence when no hook Job renders.
  • Proven in agentlab (the lab's Cilium with policyEnforcementMode: always, or the policy's rendered shape inspected against a hook Job's pod labels) before the merge.
  • CHANGELOG entry; README's hooks section says the hooks are policed by the chart.

Activity

  1. moved this from Inbox 📥 to In Progress ⛏️ in Roadmapon Sep 13, 2026
  2. moved this from In Progress ⛏️ to Validation ☑️ in Roadmapon Sep 13, 2026
  3. moved this from Validation ☑️ to Done ✅ in Roadmapon Sep 13, 2026
  4. teemow commented on Sep 13, 2026

    @teemow
    MemberAuthor

    Verified live with v4.9.1 on the first 4.x installation: the meta release upgraded from 4.7.12 to 4.9.1 on its own (Helm revision 181 deployed 2026-09-13 08:44:25Z) — details in #415.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    • Status
      Done ✅

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions