Add multi-stage build support

[emilevauge]

Hi folks,
I'm opening this issue to track multi-stage builds support in official-images.
Any ETA on this ?

[StefanScherer]

Same demand to build optimal Node.js image in nodejs/docker-node#362 as Windows isn't as good in cleaning up temp files/packages/cached MSI.

• Windows builds still use Docker 1.12.2-cs2-ws-beta (17.06.1-ee-2 is available)
• ARM builds already use Docker 17.06.1-ce
• AMD builds use Docker 17.03.2-ce
  Is there anything the community can help with?

[yosifkit]

Here is a list we came up with so far, feel free to comment with more.

Positive points:

• can generate nanoserver images where the installer exe would not normally run
• upcoming nanoserver (version 1709) based images are mostly useless without multi-stage since they have no powershell
• can download or generate a full rootfs (no more huge tarballs in git with force pushed branches)
• large build container with small end result (hello-world, traefik)

Counter points:

• can mostly be accomplished via long RUN lines
  • this will be made easier with Proposal: add "heredoc" notation to the Dockerfile syntax moby/moby#34423
• increases space required on build servers
• can greatly increase build time (busybox building binaries: 37 min for just amd64, vs a Dockerfile with ADD busybox.tar.xz /)
• updates to either base image will require a rebuild and push
• is a single binary image useful?
  • hard to debug, no shell or other utilities
  • saves about 4 MB to be from scratch vs alpine

This is more complex than just the Docker version of the systems building the images.

• how do we clean dangling images without invalidating cache of every mutli-stage image?
• hardcoded assumptions of a single FROM in many places
  • bashbrew itself
    • one example of many: in order to provide more robust build caching, bashbrew does a docker tag on an image with a hash of unique bits that includes the parent image ID
  • many jenkins jobs, shell scripts, tooling, etc used in building, tagging, pushing
    • finding instances of this assumption is difficult, but is still only half the battle; for example, we use the Architectures of the parent image of a given Dockerfile to determine which Architectures this image supports -- in a multi-stage world, that needs to instead calculate the set intersection of all the parent images' Architectures
• opinion: multi-stage build promotes messy Dockerfile since the first image isn't pushed
  • can be slightly mitigated by requiring same good practices throughout Dockerfile

[StefanScherer]

Thanks @yosifkit for the comprehensive list of pros/cons to get a better understanding of what has to be considered.
For Windows images we still need an updated Docker engine as I doubt that the current installed docker version is able to run the upcoming 1709 images properly. And then we could use multi-stage builds for reasonable nanoserver based images.
I'm used to disposable build agents like Travis/Circle/AppVeyor, so speaking about caching first stages or cleaning up dangling images is done during one build agent's life cycle, so normally it gets cleaned up after a build.
I've seen some approaches with Jenkins, but it seems that this is not a common practice.

[friism]

can greatly increase build time (busybox building binaries: 37 min for just amd64, vs a Dockerfile with ADD busybox.tar.xz /)

If you have an expedient way to build busybox, can't you just still do that? You don't have to use multi-stage builds where it doesn't make sense.

updates to either base image will require a rebuild and push

Only to the second base image, no? If the first one is the same, you could presumably re-use the first artifact. If your build-boxes are ephemeral you can cache intermediary build artifacts in a registry: moby/moby#26839

is a single binary image useful?

I think deciding whether to further slim down the slim images is a separate discussion. Even if we don't use this as an opportunity to make more images single-binary, multi-stage builds are useful.