Endless deploy loop on using before hooks which generate yaml value files

[Forestsoft-de]

My requirement is to generate the helm configuration based on certain Environment variables right before the helm deplyoment task starts.

Expected behavior

Skaffold should deploy only once and skipped paths should respected

Actual behavior

Regarding to the logs skaffold ignores my folder there a helm values file is autogenerated before the helm command is invoked
Skaffold recognize a change in the generated values file and try to redeploy the changes. The result is an endless loop of helm deployments.

Information

• Skaffold version: 1.35.2
• Operating system: Ubuntu 20.04 on wsl
• Installed via: Download
• Contents of skaffold.yaml:

deploy:
  helm:
    hooks:
      before:
          - host:
              command: ["bash", "-c", "./build/skaffold/skaffold-init.sh"]
              os: [linux]  
    releases:
      - name: my-release
        chartPath: build/charts/application
        namespace: application-dev
        upgradeOnChange: true
        skipBuildDependencies: true
        imageStrategy:
          helm: {}
        artifactOverrides:
           image: registry.gitlab.com/application
        valuesFiles:
          - ./build/skaffold/skaffold-helm-values.yaml

Steps to reproduce the behavior

1. implement a hook which regenerate the ./build/skaffold/skaffold-helm-values.yaml file on every helm run
2. skaffold dev
3. see the behavior

...
DEBU[0061] Skipping excluded path: build/skaffold
...
INFO[0062] files modified: [./build/skaffold/skaffold-helm-values.yaml] subtask=-1 task=DevLoop
...

[tejal29]

@Forestsoft-de as per the docs, can you set skipBuildDependencies: false
https://skaffold.dev/docs/pipeline-stages/deployers/helm/#helm-build-dependencies

Please let me know if it works

[Forestsoft-de]

Hey @tejal29

no its still the same problem. endless loop of deployments.

BR
Sebastian

[ericzzzzzzz]

This is caused by skaffold registers project dependencies to monitor to watch file changes before the actual first dev loop. The hook defined here modifies project dependencies when it runs and skaffold detect the change then redeploy the project and before redeploying, the hook runs again, this causes endless loop. The workaround here is to remove hooks stanza from skaffold config and run it manually.

We can perhaps to add some kind of before hooks stanza which define actions to be run before the whole config parsing, but this needs further discussion.

[haje01]

I'm experiencing the exact same problem. I'd really like to see this fixed because creating a value file gives you a lot of freedom in your Helm deployment.

[rmunn]

I'm experiencing the same issue. My skaffold pre-build hooks copy a config file into the deployment/dev folder because kustomize refuses to accept input from below its root:

manifests:
  kustomize:
    paths:
      - deployment/dev
  hooks:
    before:
      - host:
          command: ["sh", "-c", "cp otel/collector-config.yaml deployment/dev/collector-config.yaml"]
          os: [darwin, linux]
      - host:
          command: ["powershell", "-Command", "copy otel/collector-config.yaml deployment/dev/collector-config.yaml"]
          os: [windows]

This results in skaffold dev running fine the first time. But after that, any time I change a file that kustomize references, such as deployment/dev/backend-deployment.yaml, skaffold enters that infinite re-deployment loop because it keeps thinking that collector-config.yaml has changed, when it hasn't.

One possible solution would be to keep hashes of the project dependency files, and only trigger a redeploy if the hash is different; i.e. running touch somefile.yaml would not trigger a re-run. This is actually the workaround I plan to use: compare MD5 hashes before doing the cp and, if it's the same, don't run the cp command, so that skaffold won't trigger a rerun. But if skaffold could do that for me so that my before hooks don't have to get overly complex, that would be nice.

[roguh]

I use a Makefile to generate some Kubernetes manifests and I use this before-hook to run make before deploying anything. I ran into this same problem since the Makefile kept re-generating my manifest as part of the before-hook when nothing had changed, causing this loop.

I fixed this in my Makefile by leveraging make's features so that my manifest was only rebuilt if any of its dependencies had changed.

E.g. I went from

$ cat Makefile
...
manifest:
  kustomize build . > build/manifest.yaml

to

$ cat Makefile
...
build/manifest.yaml: dep1.yaml dep2.py dep3.sh
  kustomize build . > build/manifest.yaml

And I switched to using make build/manifest.yaml instead of make manifest. I also updated my other rules so that the dependencies are built in the same way, e.g. dep1.yaml could be generated from other files.