Support pre-post hooks for build, deploy and sync

[adrienjt]

Hi, in my workflow, I need to run some commands before I run skaffold. In CI/CD, I wrapped skaffold in a script, but it's a pain for developers to remember to run a script before running skaffold run or skaffold dev or for me to maintain a wrapping CLI (EDIT: and skaffold dev doesn't work, see use case below). The developer experience would be improved if skaffold supported pre-build/pre-deploy hooks (and post-build/post-deploy hooks).

Suggestion:

apiVersion: skaffold/v1...
kind: Config
hooks:
  preBuild:
    exec:
      command: ["python", "my-pre-build-hook.py"]
  postBuild:
    exec:
      command: ["./my-post-build-hook.sh"]
  preDeploy:
    exec:
      command: ["terraform", ...]
  postDeploy:
    exec:
      command: ["curl", ...]
...

Details on my use case:

• I have a Python monorepo of microservices and shared libraries. I don't want to use the root of the repo as a microservice's Dockerfile context, because of the performance hit (breaks caching, huge context to send to docker daemon or kaniko), so before I build a microservice image, I copy the required shared libraries temporarily in the microservice subfolder / Dockerfile context, and remove them afterward (Docker doesn't follow symlinks if you wonder).
• kustomize doesn't support build-time variables, so I generate overlays to label, prefix and change namespaces depending on, e.g., the pull request number or the developer name.

[davidkarim]

Would love to hear other comments regarding whether this type of feature would be desirable. In my case, I do have to add a few steps prior to running skaffold dev (such as retrieving secrets from Vault). I would definitely prefer having a way to add these pre-build hooks in the skaffold config file rather than having to build a separate wrapper script or asking developers to run a shell script prior to running skaffold dev.

[antl3x]

@dgageot we are open to ship this?

[tejal29]

@adrienjt Is it correct to say the script you have that copies the relevant/required libs to the micro-services folder is actually a way for maintaining python dependency? Or does it do something more than that?

What would a pre-deploy hook ?

@davidkarim : The steps you mentioned, like retrieving secrets from Vault, right now we expect them to be one time dev setup. Is the case where developers can't store their secret on their cluster.

[balopat]

I'm worried to make skaffold into a generic workflow execution engine and supporting the issues that come with that. Skaffold is meant to have a fast, opinionated local workflow and a building block for CI/CD use cases. Having completely arbitrary script calls starts to muddle things.

Also, the use cases mentioned here are interesting: I think I would very much so prefer helping explicitly with

• secret management
• docker context composition and
• per user / per PR manifest rendering

over leaving it to the users to implement it themselves in scripts.

[antl3x]

I'm worried to make skaffold into a generic workflow execution engine and supporting the issues that come with that. Skaffold is meant to have a fast, opinionated local workflow and a building block for CI/CD use cases. Having completely arbitrary script calls starts to muddle things.

Also, the use cases mentioned here are interesting: I think I would very much so prefer helping explicitly with

• secret management
• docker context composition and
• per user / per PR manifest rendering

over leaving it to the users to implement it themselves in scripts.

@balopat I understand your fear and I agree with you in parts. 

This local workflow that you mentioned nowadays is becoming more complex with a lot of tools that developers use during the development workflow. So, having this kind of hooks help us keep skaffold as the main entry point for all the local build/test/deployment workflow.

For example, in our case, we have some steps/scripts that should be execute before the docker build. Instead of using only the file watcher of skaffold, we are using the individual --watch flag of those scripts, because we can’t control this script execution in skaffold before the build.

[giladsh1]

@balopat @tejal29 @nthypes
I've closed #2425 in favor of this issue, but I'm copying some of the text I wrote there, in order to emphasize the need for this functionality.

IMHO These callbacks can have major befits for the development / deployment process, and make skaffold a true "one stop shop" tool.

some examples -

1. creating a beforeDev callback, which will allow the user to copy some files from a specific location before running skaffold dev. This will allow the user to copy files which are required for building the artifact, but exists in another location and can't be copied into a docker image, since they're not in the docker daemon context.

2. creating a afterDevDeployment callback, which will allow the user to run a custom command after the deployment is done when running the skaffold dev command. This action can help in many different ways, such as to create mock data for dev environments, triggers UI automation tests, and many other things. some things are not easily implemented in a docker image running somewhere.

why not use a wrapper

One could argue that these actions can be achieved with a simple wrapper before / after running skaffold. However, some actions (such as dev or debug) run in continuous mode, and can't be easily moved to the background.

[adrienjt]

@tejal29 correct, it's a way to ensure internal Python dependencies (in the same repo but outside of the Docker build context) are present in the Docker build context to be copied to the image.

I can think of two use cases for a pre-deploy hook:

1. generate manifests with anything but Helm and kustomize, e.g., Jsonnet (not currently supported by skaffold) or a custom tool (will never be supported by skaffold);
2. deploy non-Kubernetes dependencies, e.g., cloud databases, static assets, etc., with terraform, aws-cli, etc.

@balopat, I agree with @nthypes, @giladsh1 and @davidkarim that skaffold really shines when used directly rather than within a wrapper script, because of its clean developer API (something that's hard to design and maintain) and all the watch magic logic. With hooks, you'd make skaffold more usable. Therefore, more people would use it. My developer workflow is mostly supported by skaffold, but that subtle difference makes implementing skaffold a lot more complicated. Seeing what people would typically do with hooks, you could later implement "cleaner" equivalent features at your own pace.

[tbarrella]

Agreeing with what people have said above,

• Docker context composition before builds: Explicit skaffold.yaml configuration for this would be helpful, although supporting custom scripts to e.g. generate files would also be helpful to support migrations. Ideally multi-stage builds would prevent the need for generating files for the context, but not all builders have good support for multi-stage builds, such as Kaniko as of now
• Generating manifests before deploys: An example of this being helpful is when generating large numbers of variants with different parameters using a conventional programming language (see https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture/declarative-application-management.md#what-about-configurations-with-high-cyclomatic-complexity-or-massive-numbers-of-variants)