-
Notifications
You must be signed in to change notification settings - Fork 34
Migrating to charts build scripts v0.3.0
On a high-level, this migration will involve 2 PRs to release-v2.6:
- Modify Existing Scripts and Docs
- Standardize Charts, Assets, and Index (dangerous, since this modifies the released assets and index.yaml)
Once those PRs are merged, you will need to fork dev-v2.6 from release-v2.6 and make 1 more PR to "Copy Over Git History + Regenerate / Reorder Packages + Run make charts / make validate". This PR should pass GH workflow validation before being merged.
The following files need to be deleted since they are no longer used and make template does not remove them.
Run the following:
rm README.md
rm assets/README.md
rm charts/README.md
rm ./scripts/regenerate-assets
git add -A
git commit -m "Delete old README.mds and old scripts"Note: If you are pulling from a fork, you will need to modify ./scripts/version to change CHARTS_BUILD_SCRIPTS_REPO to your fork and CHARTS_BUILD_SCRIPT_VERSION to your fork first. This can be done as a temporary commit. You may also need to run git checkout -- ./scripts/version before running the git add below to avoid rewriting the change to point to a fork.
./scripts/pull-scripts
source ./scripts/version
./bin/charts-build-scripts template -r ${CHARTS_BUILD_SCRIPTS_REPO} -b ${CHARTS_BUILD_SCRIPT_VERSION}
# Note: see comment above if you are pulling scripts from a fork.
# Note: Ensure that you modify `.github/workflows/pull-request.yaml` to point to the right branches as well before continuing
git add -A
git commit -m "make template"Remove template and add validate.url and validate.branch. e.g.
helmRepo:
cname: charts.rancher.io
validate:
url: https://github.com/rancher/charts.git
branch: dev-v2.6Starting v0.3.0 of the scripts, the root README.md has been removed so that it can be customized per repository.
Instead, all docs have been relocated to a new docs/ folder.
Therefore, you will need to add a root README.md right now that is specific to your repository.
Here is an example:
## Rancher Feature Charts
This repository contains Helm charts served by Rancher Apps & Marketplace.
### Branches
- `dev-2.X` branches contain charts that under active development, to be released in an upcoming Rancher release.
- `release-v2.X` branches contain charts that have already been developed, tested, and released on an existing Rancher release.
### Making Changes
Since this repository uses [`rancher/charts-build-scripts`](https://github.com/rancher/charts-build-scripts), making changes to this repository involves three steps:
1. Adding or modifying an existing `Package` tracked in the `packages/` directory. Usually involves `make prepare`, `make patch`, and `make clean`.
2. Running `make charts` to automatically generate assets used to serve a Helm repository (`charts/`, `assets/`, and `index.yaml`) based on the contents of `packages/`.
3. [CI] Running `make validate` to ensure that all generated assets are up-to-date and ready to be merged.
#### Versioning Charts
In this repository, all packages specify the `version` field in the `package.yaml`.
The versioning scheme roughly corresponds to the following rules (with exceptions):
- Major Version: represents the Rancher minor version these charts are being released to.
- Anything less than `100`: Rancher 2.5
- `100`: Rancher 2.6
- `101`: Rancher 2.7
- etc.
- Minor Version: represents a release line of a given chart within a Rancher minor version.
- Patch Version: represents a patch to a given release line of a chart within a Rancher minor version.
As a rule of thumb, you will only ever touch this version to **increment the patch version once per Rancher patch release**. Once it has been incremented, it should not be incremented again until the next Rancher patch release, even if the chart points to an upstream that has been modified.
For more information on how package versioning works, please see [`docs/developing.md`](docs/developing.md).
#### Rancher Version Annotations
In addition to modifying the chart version, the `catalog.cattle.io/rancher-version` annotation is required for user-facing charts which show up in Rancher UI; there is no need to add the annotation to CRD charts or internal charts (like fleet).
General guidelines when releasing a new version of a user-facing chart:
1. **Ensure the chart has the annotation `catalog.cattle.io/rancher-version` with an upper bound, such as `< 2.6.99-0`**. This indicates that a fresh install of the chart should be allowed in any version of Rancher at or below `2.6.99-0` line, but should not be freshly installable in Rancher `2.7.0+`.
2. If and only if a chart will **not** work in an older version of Rancher, you will need to **amend this annotation to also include a lower bound like `>= 2.6.2-0`**. e.g. `catalog.cattle.io/rancher-version: >= 2.6.2-0 < 2.6.99-0` indicates that this chart should only be freshly installable in Rancher `2.6.2+`, but should not be freshmy installable in `Rancher 2.7.0+`.
- If you do this, it is also recommended that you **modify the previously released chart to have `catalog.cattle.io/rancher-version: < 2.6.2-0`**. For instructions on how to modify existing charts, see [`docs/developing.md`](docs/developing.md).
#### Versioning FAQ
- Do we directly backport charts to previous Rancher minor versions (e.g. make `100.x.x` available in Rancher `2.5`)?
No, we do not. If a fix needs to go to both Rancher `2.5` and `v2.6`, we just release a new chart in each branch. Then, we forward-port the one released in the `release-v2.5` branch to `release-v2.6`.
If a fix that went into Rancher `2.6` needs to be backported to Rancher `2.5`, it will be the developer's responsibility to bump the chart version in `dev-v2.5`, copy back the changes, and release a **new** chart following the Rancher `2.5` versioning scheme to `release-v2.5`.
- If Rancher `2.5` releases Monitoring `14.5.100` and `16.6.0` and Rancher `2.6` releases Monitoring `100.0.0+up14.5.100` and `100.0.1+up16.6.0`, how do we prevent users from "downgrading" from `16.6.0` to `100.0.0+up14.5.100` on a `helm upgrade` after upgrading Rancher minor versions?
Currently, this is unavoidable. There is an expectation that users should look at the upstream annotation on the chart version (e.g. `+upX.Y.Z`), read the Rancher minor version release notes, or consult the chart's `README.md` or `app-README.md` before performing an upgrade on their applications after migrating to a new Rancher minor version.
We are still looking for a better way to mitigate this kind of risk.
- For Rancher version annotations, why we don't we need to add the lower bound all the time?
Each Rancher minor version has its dedicated chart release branch (e.g. `release-v2.5`, `release-v2.6`, etc.), so a chart designed for Rancher `2.6.x` will never be available or show up in Rancher `2.5.x`; therefore, we do not need to worry about setting a lower bound of `> 2.5.99-0` on all charts.
#### Supporting Images in Airgap
Currently, the scripts used to generate the `rancher-images.txt` (used for mirroring a private registry in a air-gapped Rancher setup) rely on `values.yaml` files in charts that nest all image repository and tags used by the Helm chart under `repository` and `tag` fields.
For example:
```yaml
image: org/repo:v0.0.0 # will not be picked up
hello:
world:
# will be picked up, even though it is nested under hello.world.*
repository: org/repo
tag: v0.0.0
os: windows # optional, if this is a Windows-only image
```
Therefore, any charts that are committed into this repository must nest references to Docker images in this format within each chart's `values.yaml`; if an upstream chart you are referencing does not follow this format, it is recommended that you refactor the chart's values.yaml to look like this:
```yaml
images:
config_reloader:
repository: rancher/mirrored-jimmidyson-configmap-reload
tag: v0.4.0
fluentbit:
repository: rancher/mirrored-fluent-fluent-bit
tag: 1.7.9
fluentbit_debug:
repository: rancher/mirrored-fluent-fluent-bit
tag: 1.7.9-debug
fluentd:
repository: rancher/mirrored-banzaicloud-fluentd
tag: v1.12.4-alpine-1
nodeagent_fluentbit:
os: "windows"
repository: rancher/fluent-bit
tag: 1.7.4
```
### Links
For more information on how to make changes to this repository, please see [`docs/developing.md`](docs/developing.md).
For more information on experimental features, please see [`docs/experimental.md`](docs/experimental.md).
For more information on commands that can be run in this repository, please see [`docs/makefile.md`](docs/makefile.md).
For more information on `Packages`, please see [`docs/packages.md`](docs/packages.md).
For more information on CI, please see [`docs/validation.md`](docs/validation.md).Once it has been added, run git add README.md; git commit -m "Add repository-specific README.md"
Create a PR to merge the changes into the existing release-v2.6 branch.
An example of this PR can be found here: https://github.com/aiyengar2/charts/pull/5
NOTE: Forgot to add https://github.com/aiyengar2/charts/commit/a892df1b7bd986df03927953c0673d2149430bfe to the above PR
Important: The only changes to charts/ and assets/ should be removing the README.md and no changes should affect the index.yaml. As long as this is the case, this PR should be safe to merge without any potential impact on users.
Note: This PR can be done independently of PR 1 since it should affect a different set of files.
First, ensure that ./bin/charts-build-scripts --version outputs charts-build-scripts version v0.3.0. If this is not the case, temporarily modify ./scripts/version to point to the new version (note: if you are working with a fork of charts-build-scripts, modify both the repository and the version, where the version will be your branch name) and run ./scripts/pull-scripts. Then undo these changes and ./bin/charts-build-scripts should be at the correct version.
Before running ./bin/charts-build-scripts standardize, (which will effectively take the current contents in charts/, repackage them into assets/, unpackage the assets/ back into charts/, and regenerate the index.yaml), you will need to ensure that your current repository is valid; that is, every chart in charts/ must have a unique version.
If this is not the case, you will need to manually delete charts that have duplicate versions.
If this step is not complete, you will see something like the following when running ./bin/charts-build-scripts standardize:
$ ./bin/charts-build-scripts standardize
INFO[0000] Collecting valid charts from charts
INFO[0019] Removing collected subcharts
INFO[0019] Ensuring chart versions are unique
FATA[0019] chart longhorn at version 100.0.0+up1.1.2 is declared in charts/longhorn-1.1/longhorn/100.0.0+up1.1.2 and charts/longhorn/longhorn/100.0.0+up1.1.2 If you deleted duplicate charts, run git add -A; git commit -m "Remove duplicate charts before standardizing".
Run ./bin/charts-build-scripts standardize.
Then, if you committed resources to assets that are not charts (e.g. logos), ensure you git checkout the changes. e.g. git checkout -- assets/logos.
Once successful, run git add -A; git commit -m "make standardize".
Note: Due to the way make standardize works, it is possible that you will see minor changes to the underlying chart under charts/, namely around the ordering of the annotations in the Chart.yaml. This is because of an original bug in older versions of charts-build-scripts; even though we would always package an asset with correctly ordered annotations (due to running helm package under the hood, which does this), we would never reflect that change under charts).
Note: Due to the way make standardize works, every single TGZ file in assets will show some modifications, namely because the created timestamp has changed for the underlying file. Therefore, even though the asset has technically changed from the perspective of Git, the contents should be the same. If you would like to double check this, you can run ./bin/charts-build-scripts unzip, which will dump the contents of every single tgz in assets to charts (this is technically already run by ./bin/charts-build-scripts standardize under the hood) and you can manually inspect the changes made to charts from there to ensure nothing significant has changed.
Once the above step is done locally, create a PR to merge the changes into the new release-v2.6.
An example of this PR can be found here: https://github.com/aiyengar2/charts/pull/6
Note: I picked the wrong Longhorn chart in the Ensure that the current charts are standardizable step, which is why the Longhorn chart in this PR has removed the rancher version annotation added during an OOB release.
Important: This is the most dangerous PR of the entire migration process since it is directly modifying the structure of the assets folder and the index.yaml file. This can have potential impact on users and should be extremely carefully reviewed; at the minimum, we should manually double check that every single entry in the newly generated index.yaml covers every single expected chart that we have already released in the assets folder.
Only once the above PRs are merged, fork dev-v2.6 from release-v2.6. If dev-v2.6 already exists, rename it to something else, such as dev-v2.6-archive, as we will copy over its Git history into our new dev-v2.6 branch anyways as part of the next step.
It is recommended not to delete dev-v2.6-archive entirely since commits to the new dev-v2.6 will point back to older commits, which in turn point back to old PRs.
Make a PR to the new dev-v2.6 by running the command below (ensure that you change REMOTE to point to the Git remote you use to track rancher/charts in your local Git repository and BRANCH to point to the right branch, such as dev-v2.6 for release-v2.6):
curl https://raw.githubusercontent.com/rancher/charts-build-scripts/master/scripts/migration/cherry-pick-packages | REMOTE=upstream BRANCH=dev-v2.6-archive sh -This migration script will ensure that all the history from dev-v2.6-archive is cherry-picked into release-v2.6.
As part of the v0.3.0 script changes, Chart.yaml patches are more standardized to avoid ugly-looking patches that are difficult to rebase.
To fix the Chart.yaml patches without manual intervention, use the following steps:
- Temporarily modify
./scripts/versionto point to your old charts-build-script version (e.g.v0.2.2) - Run
./scripts/pull-scripts - Run
./bin/charts-build-scripts --version. Ensure that it outputscharts-build-scripts version v0.2.2 (XXXX) - For each chart, run
PACKAGE=<package> ./bin/charts-build-scripts prepare
- Note: you will need to run this for each chart independently due to a bug in v0.2.2 of charts-build-scripts where charts like
rancher-grafanathat are subcharts of other charts get removed on preparing the parent chart, e.g.rancher-monitoring.
- Revert
./scripts/versionchanges to point back at the new charts-build-script version - Run
./scripts/pull-scripts. - Run
./bin/charts-build-scripts --version. Ensure that it outputs the new version you expect. - For each chart in the repository, run
PACKAGE=<package> ./bin/charts-build-scripts patch- Note: this needs to be done independently instead of via a script for the same reason as above.
- Run
unset PACKAGE; ./bin/charts-build-scripts prepare(note: this affects local charts) - Run
./bin/charts-build-scripts clean.
If there are any changes, run git add -A; git commit -m "Redo Chart.yaml patches".
As part of the new v0.3.0 version of the scripts, packages can now be nested into logical groups.
While this means that the current scripts should be fully backwards compatible with the old structure of packages, it may be helpful to restructure packages at this point in order to make them more organized.
Below is an example script I ran in order to re-order packages into better logical groupings.
# Fleet
GROUP=fleet
for package in fleet fleet-agent fleet-crd; do
mkdir -p new_packages/${GROUP}/${package}
cp -R packages/${package}/* new_packages/${GROUP}/${package}
done
# Harvester
GROUP=harvester
for package in harvester-cloud-provider harvester-csi-driver; do
mkdir -p new_packages/${GROUP}/${package}
cp -R packages/${package}/* new_packages/${GROUP}/${package}
done
# Longhorn
GROUP=longhorn
for package in longhorn-1.1 longhorn-1.2; do
mkdir -p new_packages/${GROUP}/${package}
cp -R packages/${package}/* new_packages/${GROUP}/${package}
done
# AKS Operator
GROUP=rancher-aks-operator
for package in rancher-aks-operator rancher-aks-operator-crd; do
mkdir -p new_packages/${GROUP}/${package}
cp -R packages/${package}/* new_packages/${GROUP}/${package}
done
# Alerting Drivers
GROUP=rancher-alerting
for package in rancher-alerting-drivers rancher-prom2teams rancher-sachet; do
mkdir -p new_packages/${GROUP}/${package}
cp -R packages/${package}/* new_packages/${GROUP}/${package}
done
package=rancher-alerting-drivers
DEPENDENCIES_DIR=new_packages/${GROUP}/${package}/generated-changes/dependencies
yq e -i ".url = \"packages/${GROUP}/rancher-sachet\"" ${DEPENDENCIES_DIR}/sachet/dependency.yaml
yq e -i ".url = \"packages/${GROUP}/rancher-prom2teams\"" ${DEPENDENCIES_DIR}/prom2teams/dependency.yaml
# Backup
GROUP=rancher-backup
for package in rancher-backup rancher-backup-crd; do
mkdir -p new_packages/${GROUP}/${package}
cp -R packages/${package}/* new_packages/${GROUP}/${package}
done
# CIS Benchmark
package=rancher-cis-benchmark
mkdir -p new_packages/${package}
cp -R packages/${package}/* new_packages/${package}
# EKS Operator
GROUP=rancher-eks-operator
for package in rancher-eks-operator rancher-eks-operator-crd; do
mkdir -p new_packages/${GROUP}/${package}
cp -R packages/${package}/* new_packages/${GROUP}/${package}
done
# External IP Webhook
package=rancher-external-ip-webhook
mkdir -p new_packages/${package}
cp -R packages/${package}/* new_packages/${package}
# Gatekeeper
package=rancher-gatekeeper
mkdir -p new_packages/${package}
cp -R packages/${package}/* new_packages/${package}
# GKE Operator
GROUP=rancher-gke-operator
for package in rancher-gke-operator rancher-gke-operator-crd; do
mkdir -p new_packages/${GROUP}/${package}
cp -R packages/${package}/* new_packages/${GROUP}/${package}
done
# Monitoring
GROUP=rancher-monitoring
for package in rancher-grafana rancher-kube-state-metrics rancher-monitoring rancher-node-exporter rancher-prometheus-adapter rancher-pushprox rancher-windows-exporter; do
mkdir -p new_packages/${GROUP}/${package}
cp -R packages/${package}/* new_packages/${GROUP}/${package}
done
package=rancher-monitoring
DEPENDENCIES_DIR=new_packages/${GROUP}/${package}/generated-changes/dependencies
yq e -i ".url = \"packages/${GROUP}/rancher-grafana\"" ${DEPENDENCIES_DIR}/grafana/dependency.yaml
yq e -i ".url = \"packages/${GROUP}/rancher-pushprox\"" ${DEPENDENCIES_DIR}/hardenedKubelet/dependency.yaml
yq e -i ".url = \"packages/${GROUP}/rancher-pushprox\"" ${DEPENDENCIES_DIR}/hardenedNodeExporter/dependency.yaml
yq e -i ".url = \"packages/${GROUP}/rancher-pushprox\"" ${DEPENDENCIES_DIR}/k3sServer/dependency.yaml
yq e -i ".url = \"packages/${GROUP}/rancher-kube-state-metrics\"" ${DEPENDENCIES_DIR}/kube-state-metrics/dependency.yaml
yq e -i ".url = \"packages/${GROUP}/rancher-pushprox\"" ${DEPENDENCIES_DIR}/kubeAdmControllerManager/dependency.yaml
yq e -i ".url = \"packages/${GROUP}/rancher-pushprox\"" ${DEPENDENCIES_DIR}/kubeAdmEtcd/dependency.yaml
yq e -i ".url = \"packages/${GROUP}/rancher-pushprox\"" ${DEPENDENCIES_DIR}/kubeAdmProxy/dependency.yaml
yq e -i ".url = \"packages/${GROUP}/rancher-pushprox\"" ${DEPENDENCIES_DIR}/kubeAdmScheduler/dependency.yaml
yq e -i ".url = \"packages/${GROUP}/rancher-prometheus-adapter\"" ${DEPENDENCIES_DIR}/prometheus-adapter/dependency.yaml
yq e -i ".url = \"packages/${GROUP}/rancher-node-exporter\"" ${DEPENDENCIES_DIR}/prometheus-node-exporter/dependency.yaml
yq e -i ".url = \"packages/${GROUP}/rancher-pushprox\"" ${DEPENDENCIES_DIR}/rke2ControllerManager/dependency.yaml
yq e -i ".url = \"packages/${GROUP}/rancher-pushprox\"" ${DEPENDENCIES_DIR}/rke2Etcd/dependency.yaml
yq e -i ".url = \"packages/${GROUP}/rancher-pushprox\"" ${DEPENDENCIES_DIR}/rke2IngressNginx/dependency.yaml
yq e -i ".url = \"packages/${GROUP}/rancher-pushprox\"" ${DEPENDENCIES_DIR}/rke2Proxy/dependency.yaml
yq e -i ".url = \"packages/${GROUP}/rancher-pushprox\"" ${DEPENDENCIES_DIR}/rke2Scheduler/dependency.yaml
yq e -i ".url = \"packages/${GROUP}/rancher-pushprox\"" ${DEPENDENCIES_DIR}/rkeControllerManager/dependency.yaml
yq e -i ".url = \"packages/${GROUP}/rancher-pushprox\"" ${DEPENDENCIES_DIR}/rkeEtcd/dependency.yaml
yq e -i ".url = \"packages/${GROUP}/rancher-pushprox\"" ${DEPENDENCIES_DIR}/rkeIngressNginx/dependency.yaml
yq e -i ".url = \"packages/${GROUP}/rancher-pushprox\"" ${DEPENDENCIES_DIR}/rkeProxy/dependency.yaml
yq e -i ".url = \"packages/${GROUP}/rancher-pushprox\"" ${DEPENDENCIES_DIR}/rkeScheduler/dependency.yaml
yq e -i ".url = \"packages/${GROUP}/rancher-windows-exporter\"" ${DEPENDENCIES_DIR}/windowsExporter/dependency.yaml
# Istio
GROUP=rancher-istio
for package in rancher-istio rancher-kiali-server rancher-tracing; do
mkdir -p new_packages/${GROUP}/${package}
cp -R packages/${package}/* new_packages/${GROUP}/${package}
done
package=rancher-istio
DEPENDENCIES_DIR=new_packages/${GROUP}/${package}/generated-changes/dependencies
yq e -i ".url = \"packages/${GROUP}/rancher-kiali-server\"" ${DEPENDENCIES_DIR}/kiali/dependency.yaml
yq e -i ".url = \"packages/${GROUP}/rancher-tracing\"" ${DEPENDENCIES_DIR}/tracing/dependency.yaml
# Logging
package=rancher-logging
mkdir -p new_packages/${package}
cp -R packages/${package}/* new_packages/${package}
# SRIOV
package=rancher-sriov
mkdir -p new_packages/${package}
cp -R packages/${package}/* new_packages/${package}
# vSphere
GROUP=rancher-vsphere
for package in rancher-vsphere-cpi rancher-vsphere-csi; do
mkdir -p new_packages/${GROUP}/${package}
cp -R packages/${package}/* new_packages/${GROUP}/${package}
done
# Webhook
package=rancher-webhook
mkdir -p new_packages/${package}
cp -R packages/${package}/* new_packages/${package}
# Windows
package=rancher-wins-upgrader
mkdir -p new_packages/${package}
cp -R packages/${package}/* new_packages/${package}
# System Upgrade Controller
package=system-upgrade-controller
mkdir -p new_packages/${package}
cp -R packages/${package}/* new_packages/${package}
rm -rf packages
mv new_packages packages
After this is run, ensure that you git add packages; git commit -m "Restructure packages into logical groups". You should observe that all the changes are just renames.
Run ./bin/charts-build-scripts charts to generate charts for all current packages.
Then run git add -A; git commit -m "make charts"
Run ./bin/charts-build-scripts validate; it is expected that this command should fail and a release.yaml should be outputted.
Then run git add -A; git commit -m "Add release.yaml based on existing packages"
Run ./bin/charts-build-scripts validate once more; ensure that it passes.
Run ./scripts/package-ci as a final check that all the basic Makefile commands pass on this repository.
Once the above step is done locally, create a PR to merge the changes into dev-v2.6.
An example of this PR can be found here: https://github.com/aiyengar2/charts/pull/7
Note: I picked the wrong Longhorn chart in the Ensure that the current charts are standardizable step, which is why this PR shows the make charts commit changing the index.yaml, release.yaml, and chart / asset for Longhorn 1.1. This is not expected.
Important: You should ensure that CI is passing before approving / merging this PR.