AutoPull Images in cacheFrom Should not be default behavior

[adawalli]

Expected behavior

PR #1495 introduced pre-pulling behavior as the default when the cacheFrom keyword is used (with no option to disable this, I believe).

In my humble opinion, this is a mistake and honestly can introduce security risks.

Let me explain. If the first or second command of a Dockerfile is something like apt-get update && apt-get dist-upgrade -y, the user might think that they are actually getting the latest security updates periodically. Now, let's say the dockerfile only changes towards the end (changes default shell for example). In skaffold's current behavior, if I want to use cacheFrom to speed up builds, it downloads the latest image, which from a Dockerfile perspective only shows a change on the last line. it's possible that another security update will never be run again until the cache images are invalidated!

The way we previously handled this with our docker builds in gitlab runners was something like this

docker pull <image>:latest
docker image prune -a --force --filter "until=12h"
docker build --cache-from <image>:latest .....

In this way, we would invalidate the cache anytime it had cooled off for more than 12 hours. With the current implementation of skaffold, this isn't really possible.

The following file will always pull the cache even with tryImportMissing: false (which is likely an unrelated command, but confusing all the same)

apiVersion: skaffold/v2beta24
kind: Config
build:
  artifacts:
    - image: myimage
      context: .
      docker:
        cacheFrom:
          - "myimage:latest"
  local:
    useBuildkit: true
    push: false
    concurrency: 0
    tryImportMissing: false

A couple of possible solutions from (my) preferred to least preferred

• Skaffold remove this "always pull" cacheFrom behavior as the default policy with an option to enable it
• Skaffold leave the default behavior as is, but allow an option to disable it on a per image basis
• evaluate tryImportMissing ...and don't import cache files if they are...missing
• Use Patching to remove cacheFrom stanzas conditionally based on some variable - this is painful and devs are likely going to forget to set variables for this
• Add an environment variable near the top of the Dockerfiles with the date, that changes every 24 hours, and thus invalidates the cache - this is error prone since devs will forget to put this in their files. Also, we will now waste time pulling a cache image that we might know is invalid.
• Create a secondary service that intentionally invalidates my cache every night (e.g., docker pull scratch && docker tag scratch <myimage>:latest && docker push <myimage>:latest) - This is getting pretty gross in my opinion
• Stop using caching completely - Unfortunately, we can't afford to do this...our images are time-consuming to build

Information

• Skaffold version: v1.35.0
• Operating system: Ubuntu, 18.04.6 LTS
• Installed via: skaffold.dev

[adawalli]

@dgageot FYI

[adawalli]

@briandealwis I believe this one should be considered a fairly high priority as it potentially introduces poor security postures into folk's images - and they will likely have no idea.

[tejal29]

@adawalli Thanks for the issue and the explaining the security concerns here. I like your top 2 of your preferred options i.e. enable or disable it via config for two reasons

1. In dev loop, developers do want the subsequent builds to be faster.
2. For CI/CD purposes, skaffold should always build fresh or users should have an option invalidate cache after X mins.

We will look into this as part of efforts to enable users to securely build software with best security practices.

[cyraid]

How's this one coming along? Did something change since? Personally I like the second one as I'm coming up with instances where I'd like to invalidate the cache of only one artifact.

Being able to name an artifact, and reference it like:

artifacts:
- name: api-server
  image: some.registry/url/api-server
- name: web-server
  image: some.registry/url/web-server

skaffold invalidate -p release api-server

I believe you can currently do:

skaffold build -b some.registry/url/api-server -p release --cache-artifacts=false

But who can remember that? ^_^