Allow for building an image with multiple --cache-from images

[richafrank]

We're using this to --cache-from intermediate and final stages of a multi-stage docker build, instead of busting the cache every time on building the intermediate target.

Would definitely appreciate a review of the interface and implementation.

[irlevesque]

This is a huge win for anyone building multi-stage images that want to use intermediate cache for faster builds.

[irlevesque]

@lox / @toolmantim Quantopian has been using this internally for a while and would love to see it merged upstream. I think other customers would certainly benefit. Anything holding you back that we can address?

[toolmantim]

@irlevesque sorry for the delay here, and thanks for the thorough PR. It's definitely something we'd love to support — I'll get this reviewed for you, we'd love to get it merged.

[richafrank]

Thanks so much @toolmantim .

[toolmantim]

I can definitely see a need for this… even though it's starting to create some pretty wild bash code.

Did you consider some other alternative syntaxes to the readme, rather than adding another layer of : use?

I ask because setting up cache-from properly is something I'd love every new BK user to setup from the get go, and avoid cold caches as much as possible… and I think many people use Docker targets.

[toolmantim]

I was wondering if we could provide a more realistic example for this one, because I think it's intended on having the build number interpolated into it too?

I'm just trying to wrap my head around it, from the perspective of a user reading the readme, who wants to know "What sort of thing should I copy+paste into my pipeline.yml file?"

[richafrank]

Ah, fair point about the interpolation.

A better example would be to show an intermediate target build step as well and how the two steps tie together:

  - label: ":docker: Build Intermediate Image"
    plugins:
      - docker-compose#v3.2.0:
          build:
            - myservice_intermediate  # docker-compose.yml is the same as myservice but has `target: intermediate`
          image-name: buildkite-build-${BUILDKITE_BUILD_NUMBER}
          image-repository: index.docker.io/myorg/myrepo/myservice_intermediate
          cache-from:
            - myservice_intermediate:index.docker.io/myorg/myrepo/myservice_intermediate:${BUILDKITE_BRANCH}
            - myservice_intermediate:index.docker.io/myorg/myrepo/myservice_intermediate:latest
          push:
            - myservice_intermediate:index.docker.io/myorg/myrepo/myservice_intermediate:${BUILDKITE_BRANCH}

  - wait

  - label: ":docker: Build Final Image"
    plugins:
      - docker-compose#v3.2.0:
          build:
            - myservice
          image-name: buildkite-build-${BUILDKITE_BUILD_NUMBER}
          image-repository: index.docker.io/myorg/myrepo
          cache-from:
            - myservice:index.docker.io/myorg/myrepo/myservice_intermediate:buildkite-build-${BUILDKITE_BUILD_NUMBER}:intermediate  # built in step above
            - myservice:index.docker.io/myorg/myrepo/myservice:${BUILDKITE_BRANCH}
            - myservice:index.docker.io/myorg/myrepo/myservice:latest
          push:
            - myservice:index.docker.io/myorg/myrepo/myservice:${BUILDKITE_BRANCH}
            - myservice:index.docker.io/myorg/myrepo/myservice:latest

[richafrank]

Thanks for the feedback!

I can definitely see a need for this… even though it's starting to create some pretty wild bash code.

Definitely agreed - please feel free to suggest better bash-isms. I was mostly extending what was here, which required some new bash learnings on my part.

Did you consider some other alternative syntaxes to the readme, rather than adding another layer of : use?

My sense of the requirement here is that we need a bit more hierarchy in order to group cache-froms together.

• I went with adding an extra tag to each, but only because that was easiest for me to parse from the yaml. Changing the structure of the yaml was my first thought, but I wasn't aware of existing tooling à la plugin_read_list to parse, say, a dictionary. Let me know what you recommend.
• I chose the : for the tag only because I knew it wouldn't be in use in the repo/image name. If we end up keeping this tag-based design, definitely let me know if there's a better tag.

[pzeballos]

Hey @richafrank! Sorry for the late reply! 🙏🏻 We are checking all the PRs open we have, and we are wondering if you are still interested in this PR and if are willing to update it and maybe do the changes that Tim suggested. We can close this one and you can create a new one. Thanks!

[richafrank]

Hi @pzeballos ! My need is somewhat stale, but I'd be happy to help push this forward if you're interested in it.

• I can update the README with my reply above.
• For the rest, I had replied to Tim's comments needing more info. Would you be able to advise there? Thanks!

[nsuma8989]

@richafrank We discussed with Tim and we are good with the changes. So if you can update the README with changes you mentioned in the reply and resolve the conflicts we can go ahead with merging the change.

[nsuma8989]

@richafrank Just following up to see if you had a chance to look into above comment

[richafrank]

Hi @nsuma8989 , thanks for the reply! Sounds good to me, but I won't be able to get to it this week.

[nsuma8989]

@richafrank Thank you for the response. Yes, it does not need to be this week and you can just leave a comment whenever you are done.

[richafrank]

@nsuma8989 How might I trigger a remote buildkite build à la https://buildkite.com/buildkite/plugins-docker-compose/builds/ , to see if the changes are still ok? I tried opening a new PR, but that didn't work either.

[pzeballos]

Hey @richafrank! We have disabled the option to trigger new builds for open PR and third-party repos (we'll check this out). The other PR #323 is ready? Can you close this one so we'll continue on the other one?
I did run a build on #323 (which failed 😅), but now all subsequent commits should trigger a new build 🙂
Thanks!

[toote]

Closing this PR as it is the same (and contained in) #323