Skip to content

feat(network): support Kustomize-style patches for the Cilium feature - #90

Draft
louiseschmidtgen wants to merge 1 commit into
mainfrom
feature/kustomize-patches-network
Draft

louiseschmidtgen wants to merge 1 commit into
mainfrom
feature/kustomize-patches-network

Conversation

@louiseschmidtgen

Copy link
Copy Markdown
Contributor

What

Implements the "typed Patches field" design for enabling Kustomize-style manifest customization on the network (Cilium) feature, without exposing Helm or Kustomize to end users. See the linked API PR (canonical/k8s-snap-api#71) for the public type definitions.

k8s set network.patches='
- target:
    kind: DaemonSet
    name: cilium
  strategic-merge: |
    spec:
      template:
        spec:
          containers:
          - name: cilium-agent
            resources:
              limits:
                memory: 2Gi
'

How

  • pkg/client/helm: a new in-process Kustomize post-renderer (NewKustomizePostRenderer) is wired into Client.Apply via Helm's native PostRenderer hook (action.Install.PostRenderer / action.Upgrade.PostRenderer). No files are written to disk and no kustomize/helm binary is shelled out to — everything runs through sigs.k8s.io/kustomize/api against an in-memory filesystem (kyaml/filesys.MakeFsInMemory).
    • Patch targets are pre-validated against the actually-rendered manifest before kustomize runs. This closes a real kustomize gap: a JSON6902 patch against a target that doesn't exist silently no-ops instead of erroring (confirmed by reading PatchTransformer's source and reproducing it in a test) — we now surface a clear error instead.
    • A SHA-256 digest of the patches is folded into the values map used by Client.Apply's existing change-detection logic (sameValues/sameVersions), under a reserved key. Without this, a patch-only change would be invisible to that diff (since patches are applied out-of-band, never part of chart values) and the upgrade would be silently skipped.
    • Client.Apply gained a variadic patches ...Patch parameter, so all ~19 existing call sites (metrics-server, metallb, coredns, cilium's other appliers, localpv, ...) remain source-compatible without changes.
  • pkg/k8sd/types: internal Patch / PatchTarget types and a Network.Patches *[]Patch field, wired through the exact same plumbing every other ClusterConfig field goes through: ClusterConfigFromUserFacing / ToUserFacing (API <-> internal conversion, with inline validation), mergeSliceField (bootstrap/join config merge), and ClusterConfig.Validate() (target kind/name required, exactly one of strategic-merge/json6902).
  • pkg/k8sd/features/cilium: ApplyNetwork now threads network.GetPatches() into the Helm client's Apply call via a small toHelmPatches conversion helper.
  • cmd/k8s: k8s set network.patches=<yaml> / k8s get network.patches wiring. Since the existing mapstructure decode hooks only handle []string and map[string]string, a new hook (utils.YAMLToMapSliceHookFunc) YAML-decodes the CLI value into []map[string]any so mapstructure can finish decoding it into []apiv2.Patch.

Why this design (vs. raw kustomize/kubectl-patch passthrough)

Full design discussion happened outside this PR; the short version: a typed field keeps validation, docs, and the k8s get/set UX consistent with every other feature knob, and avoids ever exposing kustomize's full schema (bases, generators, transformers, etc.) or requiring users to know a kustomization.yaml even exists. The trade-off is that only the two most common kustomize patch primitives (strategic-merge, JSON6902) are supported — no bases/components/generators.

Known, load-bearing limitation

Only regular rendered manifests pass through Helm's PostRenderer. Resources annotated with helm.sh/hook (e.g. Cilium's cleanup hook Job) never do, and can't be targeted by a patch. This is a Helm SDK architectural constraint, not something this implementation can work around.

Scope

Only the network/Cilium feature is wired up as a first vertical slice. If this design is accepted, DNS, ingress, gateway, load-balancer, and local-storage can follow the same pattern in follow-up PRs.

Dependency on canonical/k8s-snap-api#71

go.mod currently points at the feature/network-patches-field branch of k8s-snap-api via a pseudo-version, so this PR can be built and reviewed independently. This will be updated to point at a tagged release once that PR merges, per the repo's documented multi-repo dev convention (see AGENTS.md).

New dependencies

go mod tidy promoted sigs.k8s.io/kustomize/api, sigs.k8s.io/kustomize/kyaml, and sigs.k8s.io/yaml from indirect to direct dependencies. No new transitive dependencies were introduced — all three were already present in go.sum (pulled in transitively by existing dependencies).

Testing

  • pkg/client/helm: fully built and tested locally (go build, go test -v). 5 new tests cover no-patches passthrough, strategic-merge, JSON6902, an unknown-target error case, and patch validation (3 subcases).
  • pkg/k8sd/types (convert/validate/merge) and pkg/utils (new mapstructure hook): could not be locally built/tested in this sandbox — these packages transitively import github.com/canonical/go-dqlite/v3 (requires CGO + native dqlite headers) and github.com/canonical/lxd (Linux-only syscall constants) via pkg/utils -> microcluster. Confirmed via git stash that this is a pre-existing, environment-only limitation on this (macOS) sandbox, not caused by this change. Multipass and LXD were both tried as alternatives to Docker per review guidance, but neither is usable here (multipass: daemon has stale/invalid certs and requires sudo to fix; LXD: no remote/server configured, and LXD cannot run natively on macOS in the first place).
    • As a substitute, the core logic (patchesFromAPI / patchesToAPI / validatePatches, and the new YAMLToMapSliceHookFunc mapstructure hook) was copied into standalone scratch Go modules with no dqlite/lxd dependency and verified there with passing tests, in addition to careful manual review against the existing ClusterConfig field conventions (e.g. KubeProxyEnabled, UpstreamNameservers).
    • Unit tests for all of this are included in the diff (cluster_config_validate_test.go, cluster_config_convert_patches_test.go, cluster_config_merge_test.go, mapstructure_test.go) and should be treated as the source of truth once CI runs them on Linux.

Follow-ups (not in this PR)

  • Wire additional features (DNS, ingress, gateway, load-balancer, local-storage) onto the same Patches pattern.
  • Update user-facing docs once the design is confirmed.

Adds a typed `Patches` field to the network feature (`k8s set
network.patches=...`) that lets operators customize the rendered Cilium
Helm manifest without exposing Helm or Kustomize to end users.

- pkg/client/helm: a new in-process kustomize post-renderer
  (NewKustomizePostRenderer) is wired into Client.Apply via Helm's native
  PostRenderer hook. Patches never touch disk or shell out to a kustomize
  binary; everything runs through sigs.k8s.io/kustomize/api in memory.
  Patch targets are pre-validated against the rendered manifest to avoid
  a kustomize JSON6902 gap where a non-matching target silently no-ops
  instead of erroring. A patches digest is folded into the values used
  for Helm's existing change-detection, so patch-only changes are not
  silently skipped on upgrade.
- pkg/k8sd/types: internal Patch/PatchTarget types, wired through the
  existing convert/merge/validate plumbing that every other ClusterConfig
  field goes through (ClusterConfigFromUserFacing/ToUserFacing,
  mergeSliceField, ClusterConfig.Validate).
- pkg/k8sd/features/cilium: ApplyNetwork now threads
  network.GetPatches() into the Helm client's Apply call.
- cmd/k8s: 'k8s set network.patches=...' / 'k8s get network.patches'
  wiring, including a new mapstructure decode hook
  (YAMLToMapSliceHookFunc) to parse a YAML list of patch objects from
  the CLI's existing key=value flag format.
- Requires the companion canonical/k8s-snap-api PR (Patches field on
  NetworkConfig); go.mod currently points at that branch via a
  pseudo-version and will be updated to a tagged release before merge.

Only the Cilium/network feature is wired up as a first vertical slice;
other features (DNS, ingress, gateway, load-balancer, local-storage)
can follow the same pattern.

Known limitation: this sandbox cannot locally build/test
pkg/k8sd/types, pkg/k8sd/features/cilium, or cmd/k8s, because they
transitively import github.com/canonical/go-dqlite/v3 (requires CGO +
native dqlite headers) and github.com/canonical/lxd (Linux-only syscall
constants) via pkg/utils -> microcluster. This is a pre-existing,
environment-only limitation verified via git stash on vanilla code, not
caused by this change. Multipass and LXD were both attempted as
alternatives to Docker but are non-functional in this sandbox (multipass:
stale daemon certs requiring sudo; LXD: no remote/server configured, and
LXD cannot run natively on macOS). The affected logic (patchesFromAPI/
patchesToAPI/validatePatches, and the new mapstructure hook) was verified
independently in isolated scratch Go modules with the same logic copied
out from CGO/dqlite dependencies, in addition to careful manual review
against existing ClusterConfig field conventions. CI is the authoritative
build/test signal for these packages.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: louiseschmidtgen <louise.schmidtgen@canonical.com>
@louiseschmidtgen

Copy link
Copy Markdown
Contributor Author

Companion API PR: canonical/k8s-snap-api#71

@louiseschmidtgen

Copy link
Copy Markdown
Contributor Author

k8s-snap version-pin PR for building/testing this branch: canonical/k8s-snap#2838

@bschimke95

Copy link
Copy Markdown
Collaborator

@louiseschmidtgen is this still relevant or should we close it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants