Proposal: Helm-First Resource Definitions

[c-kruse]

This proposal makes the Skupper Helm chart the source of truth for deployment resources (Deployment, RBAC, ServiceAccount, etc.), replacing the previous source of truth embedded in the skupper-deployment-generator.sh script, operator bundle generator script, and config/rbac kustomize directory.

Motivation

• Clarity: The source of truth for our resources is presently fractured: deployments and rbac resources are embedded in a shell script, and are duplicated for the operator bundle build.
• Maintainability: Injecting templates for helm into a plain YAML resources is difficult.
• Flexibility: Users have requested additional deployment options (tolerations, affinity, resource limits) and will eventually need guidance with alternate controller configurations (access types, grants server config, etc.)

Expands on #2351

Changes

Architecture

├── charts
│   ├── skupper
│   │   └── Chart.yaml // Source of truth for deployment resources
│   └── skupper-crds
│       └── Chart.yaml // Generated crds-only chart (optional)
└── config
    ├── crd
    │   └── kustomization.yaml // Source of truth for CRD definitions
    └── samples
        └── kustomization.yaml // Source of truth for examples for operator bundle

• Add a source-controlled skupper chart
• Remove crds from skupper chart
• Refactor the deployment-generator script to use helm template and kustomize to produce install manifests
• Refactor the operator-bundle-generator script to use helm template, kustomize and operator-sdk to generate bundles.
• Extend skupper chart with community requested template values: tolerations, affinity, resources, etc.
• Source of truth for CRDs and samples remains in config tree.
• Add a crds-only manifest - generated with kustomize ./config/crd
• Add a skupper-crds Chart with templates generated from crds.

Artifacts

Add:

• skupper-crds.yaml For cluster admins to install CRDs independently from a manifest file.
• oci://quay.io/skupper/helm/skupper-crds CRD only helm chart functionally identical to skupper-crds.yaml. For cluster admins to install CRDs independently but with helm involved.

Revised:

• oci://quay.io/skupper/helm/skupper skupper helm chart - backwards compatible with exception for the removal of CRDs.

Alternative Approaches

Kustomize-First

An alternative approach would be to make kustomize the canonical source by moving deployment specs into ./config a la kubebuilder and operator framework. The Helm chart would either continue to be generated, or resource definitions would be duplicated.

Pros	Cons
No templates - better IDE support	Helm chart requires continued generation or duplicate resource definitions
Removes Helm as a development dependency	Extending helm templates requires duplicate resource definitions
Convenient for operator builds	Overlays can be as difficult as templates

Why Helm-First

This proposal chooses Helm-first because:

• Helm is widely adopted for Kubernetes
• Maintaining resource templates is easier than maintaining parity between resource templates and plain yaml resources

Why keep CRDs separate

• CRDs are a cluster-global resource, and removing or downgrading CRDs can result in data loss.
• The skupper API is growing, the skupper helm chart using the special crds directory does not support upgrades, downgrades or uninstalls. It follows that users of the current skupper helm chart will need to manually intervene as the skupper api grows.
• A separate skupper-crds chart that treats CRDs as regular resources (templates) can be upgraded when needed, as an alternative to sourcing the latest crds from a manifest. Appropriate warnings can be made about downgrading, and helm resource-policy annotations can prevent chart uninstall from removing the CRDs.

[c-kruse]

@ajssmith @nluaces @fgiorgetti would like to get your thoughts here.

Have a PoC that spits out (very nearly) byte-identical deployment manifests from the chart.

I think the operator bundle script will also go fairly easily. The only friction point I anticipate is for the downstream operator build scripts.

[nluaces]

Very good analysis!
How is the skupper-crds helm chart going to be generated?

[c-kruse]

@nluaces what I had looked something like this:

• We keep the skupper-crds Chart.yaml, readme and (empty) values.yaml in source and .gitignore the templates.
• Add a kustomize overlay (either something we actually keep in source or generate in a script)

mkdir -p ./config/hack/skupper-crds-helm-templates; cat << 'EOF' > ./config/hack/skupper-crds-helm-templates/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../crd
commonAnnotations:
  helm.sh/resource-policy: keep
labels:
- pairs:
    app.kubernetes.io/name: skupper-crds
EOF

• Add a make generate target that does something like this:

rm ./charts/skupper-crds/templates/*
kubectl kustomize ./config/hack/skupper-crds-helm-templates/ > ./charts/skupper-crds/templates/skupper-crds.yaml

[fgiorgetti]

I really like the overall standardization proposal using templates.

oci://quay.io/skupper/helm/skupper skupper helm chart - backwards compatible with exception for the removal of CRDs.

This change must be well documented as part of the release notes and an important section must be provided, explaining the skupper-crds must be installed to preserve original behavior.

The only friction point I anticipate is for the downstream operator build scripts.

It was the first thing that came to my mind. But I believe it won't be a big deal once we have it working fine upstream.