Skip to content

Add a smoke/full addon profile so dev spokes can skip the heavy catalog #83

Description

@stxkxs

Context

#82 ("Full addon catalog overwhelms small/dev clusters") was substantially resolved by three infra fixes (Cilium prefix-delegation for ~4x node IPs, the NodePool arch fix, and a system-cluster-critical Karpenter priority). Prefix-delegation removes the IP-exhaustion root cause directly. This issue tracks the remaining, separable optimization from that ticket: a profile knob so a throwaway dev spoke doesn't install the full catalog at all.

What

A cluster-secret label addon_profile (smoke | full, default full) that gates the heavy, optional, spoke-targeting addons:

  • Heavy/optional on spokes: falco, trivy, gpu-operator, aws-neuron, the grafana/prometheus stack.
  • Always-on (critical): cert-manager, external-secrets, cilium, karpenter, the operator, kyverno policies.

ApplicationSet cluster generators gate with a safe-by-default expression — a cluster with no label still matches, so prod is unaffected:

matchExpressions:
  - key: addon_profile
    operator: NotIn
    values: [smoke]

The wrinkle

addons-security bundles kyverno (policy enforcement — should stay) with falco + trivy (heavy — should skip). Appset-level gating is all-or-nothing, so this needs the security appset split into a policy appset (always-on) and a scanning appset (profile-gated) rather than a single matchExpression.

Companion (landing-zone)

The gate is inert until something sets the label. landing-zone cluster-bootstrap should stamp addon_profile on the cluster secret from the environment (dev → smoke, staging/prod → full), or expose it as a Cluster claim field in eks-fleet.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions