Helm release is not updated if chart version has changed in skaffold config file

[dnovvak]

Expected behavior

The command skaffold deploy --skip-render upgrades a release to a new version of a chart if deploy.helm.releases.[].version has changed.

Actual behavior

The command skaffold deploy --skip-render does nothing if deploy.helm.releases.[].version has changed.

Information

• Skaffold version: 1.27.0
• Operating system: Windows 10
• Installed via: binary downloaded from GitHub release
• Contents of skaffold.yaml:

apiVersion: skaffold/v2beta18
kind: Config
metadata:
  name: cloud-platform-k8s
deploy:
  helm:
    flags:
      upgrade:
      - --install
    releases:
    - name: nginx-ingress
      namespace: ingress-nginx
      createNamespace: true
      repo: https://kubernetes.github.io/ingress-nginx
      remoteChart: ingress-nginx
      version: 3.33.0
      valuesFiles:
      - controllers/ingress-nginx/values.yaml
      wait: true

Steps to reproduce the behavior

1. Use the above skaffold config (valuesFiles may be skipped)
2. skaffold deploy --skip-render
3. Verify installed chart version:

> helm list --all-namespaces
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
nginx-ingress   ingress-nginx   1               2021-06-30 11:17:40.6111103 +0200 CEST  deployed        ingress-nginx-3.33.0    0.47.0

4. Change the deploy.helm.releases.[].version to 3.34.0 and repeat step 2 and 3
5. The version is still 3.33.0

Deboug output

> skaffold deploy --skip-render -v debug
time="2021-06-30T12:19:19+02:00" level=info msg="Skaffold &{Version:v1.27.0 ConfigVersion:skaffold/v2beta18 GitVersion: GitCommit:1f46f249c832bb2a99e3285ad327647c95ff4bb9 BuildDate:2021-06-29T21:40:35Z GoVersion:go1.14.14 Compiler:gc Platform:windows/amd64 User:}"
time="2021-06-30T12:19:19+02:00" level=info msg="Loaded Skaffold defaults from \"C:\\\\Users\\\\user\\\\.skaffold\\\\config\""
time="2021-06-30T12:19:19+02:00" level=debug msg="parsed 1 configs from configuration file C:\\Git\\cloud-platform-k8s\\skaffold.yaml"
time="2021-06-30T12:19:19+02:00" level=debug msg="Defaulting build type to local build"
time="2021-06-30T12:19:19+02:00" level=info msg="Using kubectl context: <context>"
time="2021-06-30T12:19:19+02:00" level=debug msg="Running command: [minikube version --output=json]"
time="2021-06-30T12:19:19+02:00" level=debug msg="setting Docker user agent to skaffold-v1.27.0"
time="2021-06-30T12:19:19+02:00" level=debug msg="Using builder: local"
time="2021-06-30T12:19:19+02:00" level=debug msg="push value not present in NewBuilder, defaulting to true because cluster.PushImages is true"
time="2021-06-30T12:19:19+02:00" level=info msg="build concurrency first set to 0 parsed from *local.Builder[0]"
time="2021-06-30T12:19:19+02:00" level=info msg="final build concurrency value is 0"
time="2021-06-30T12:19:19+02:00" level=debug msg="Running command: [helm version --client]"
time="2021-06-30T12:19:19+02:00" level=debug msg="could not parse date \"\""
time="2021-06-30T12:19:19+02:00" level=debug msg="Command output: [version.BuildInfo{Version:\"v3.5.4\", GitCommit:\"1b5edb69df3d3a08df77c9902dc17af864ff05d1\", GitTreeState:\"clean\", GoVersion:\"go1.15.11\"}\n]"
Tags used in deployment:
time="2021-06-30T12:19:19+02:00" level=debug msg="getting client config for kubeContext: ``"
Starting deploy...
time="2021-06-30T12:19:20+02:00" level=info msg="Deploying with helm v3.5.4 ..."
time="2021-06-30T12:19:20+02:00" level=debug msg="Executing template [...]
time="2021-06-30T12:19:20+02:00" level=debug msg="Running command: [helm --kube-context <context> get all --namespace ingress-nginx nginx-ingress]"
time="2021-06-30T12:19:21+02:00" level=info msg="Release nginx-ingress not upgraded as it is remote..."
time="2021-06-30T12:19:21+02:00" level=info msg="Deploy completed in 969.6978ms"
Waiting for deployments to stabilize...
time="2021-06-30T12:19:21+02:00" level=debug msg="getting client config for kubeContext: ``"
Deployments stabilized in 550.399ms

time="2021-06-30T12:19:22+02:00" level=debug msg="exporting metrics"
time="2021-06-30T12:19:23+02:00" level=debug msg="metrics uploading complete in 967.2939ms"

[tejal29]

/cc @marlon-gamez to link this on his epic.

[briandealwis]

@dnovvak Skaffold does have an upgradeOnChange setting that defaults to false for remote charts. I've been unable to verify that it should work for this case as ingress-nginx fails when I try to install it, and I haven't had time to figure out why 😠

[dnovvak]

@briandealwis Thank you for the hint.

Indeed, with upgradeOnChange: true I was able to perform the upgrade.

There is a drawback of this solution however - running deploy once again with no changes in skaffold.yaml file containing upgradeOnChange: true causes forced redeploy and new revision in helm releases.

An example with cert-manager chart:

> helm list --all-namespaces
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                    APP VERSION
cert-manager    cert-manager    2               2021-10-05 18:09:09.6453503 +0200 CEST  deployed        cert-manager-v1.5.3      v1.5.3

and after skaffold deploy --skip-render without any changes in skaffold.yaml a new revision has been created:

> helm list --all-namespaces
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                    APP VERSION
cert-manager    cert-manager    3               2021-10-05 17:30:45.595010284 +0000 UTC deployed        cert-manager-v1.5.3      v1.5.3

My expectation is to use skaffold in stable and declarative way.

[ktarplee]

This is still the case in skaffold v2.13.2. This is basically a blocker for us since I cannot reliably deploy helm releases when the skaffold.yaml file changes (e.g., the chart version in this case).