Why are COPY layers cacheable?

[isker]

The README says that only RUN layers are cacheable. But last year COPYs were also made cacheable
in dbabcb1 (cc @donmccasland).

I'm not an expert in docker, but it's not clear at face value why you'd want to cache such layers. If you're COPYing some data in, you already have it locally, right? But when layer caching is enabled with a remote repository, instead of just executing that local copy, you hash that local data, then go to the network to download a layer containing the same data. Or, upon a miss, you spend time pushing the layer to your cache.

We have a performance-sensitive image build using Kaniko that has a few RUNs that we'd like cached, and some large COPYs that, being cached, are actually offsetting performance improvements from caching RUNs. And our image cache is much larger than we'd like.

@donmccasland can you clarify the intent of cacheable COPYs? And would you be receptive to a PR adding a flag to disable caching for COPY commands?

Thanks for your time!

[kamaln7]

My guess is that this was added for non disk-local contexts like S3 & friends. If that is the case maybe disabling COPY caching when the context is a dir:// or tar:// would be a good balance?

[isker]

As far as I can tell from reading the docs, you can only COPY things that are already in the local docker build context.

Even if they could be remote, you'd still have to hash them in order to check the layer cache. And to hash them you'd presumably have to download them locally first.

[isker]

@donmccasland sorry to ping you again, but if you can provide any guidance here I'd appreciate it! If there's anything to do I'm more than happy to implement.

[tejal29]

@isker Sorry for the delay. @donmccasland is no longer actively working on this project.

Reason why copy layers are cacheable is to determine if the subsequent run commands can be used from cache.
e.g.
in order to determine if the subsequent Run command can be used from cached layer, kaniko computes if package.json has changed. if it has changed, then cache layer for Run yarn install is invalidated.
If not, then we use the cache layer for Run command.

Copy package.json . 
Run yarn install

The way kaniko caching works is, it uses the cache until a cache key for one command in dockerfile is changed.
once it detects a change, it stops using cache for subsequent run commands.

I would suggest, you could re-arrange your dockerfile so that file copies not affecting run command should come later.
Doed that make sense?

[isker]

@tejal29 Thanks for responding.

The way kaniko caching works is, it uses the cache until a cache key for one command in dockerfile is changed.
once it detects a change, it stops using cache for subsequent run commands.

This is what I did not understand. Unfortunately our Dockerfile is fundamentally copy-heavy. The ordering of COPYs and RUNs is already optimal in this regard.

If you'll entertain more questions, why does Kaniko's layer cache work that way? Why can't it go back to the cache after producing a layer locally?

I will open a PR documenting this behavior in the README.

[kamaln7]

Thanks for the explanation @tejal29 that makes sense!

I'm trying to follow the caching logic here and I'm probably missing something obvious, but would something like this work?

With COPY commands specifically, instead of:

1. calculate a composite key
2. attempt to retrieve a layer with the same key
3. if:
   • found: replace the COPY command with a "fake" one that applies the cached layer instead
   • not found: stop the caching process, keep the COPY command as-is.
     • later after it's executed, upload the new cached layer to the registry with the new key for future builds

do something like:

1. calculate a composite key
2. check if a remote layer with the same key exists - but do not retrieve it
3. if:
   • found: keep the COPY command as-is, don't stop the caching process
   • not found: keep the COPY command as-is, stop the caching process
     • later after it's executed, upload the an empty layer to the registry with the new key for future builds

This way the local context is always used as a source for COPY commands, but would also be used to determine whether the existing cache for subsequent layers can be used or not.

Edit: thinking about it some more, we don’t even need to upload an empty layer or check for one’s existence. Just include the COPY command’s key in the subsequent commands’ composite keys-which is already done?

[tejal29]

@isker we can do in a person sync sometime if that helps.
I understand that the image cache size issue and would love to speed up builds for Copy heavy dockerfiles.

I like @kamaln7's solution to not stop caching where

1. calculate the composite key for copy command but only use to seed subsequent Run commands cache key.
   All copy command will be executed from local sources
2. for Run command, if remote layer with the same key is present, retrieve the layer
3. if remote layer with the same key does not exist, stop caching subsequent commands.

@kamaln7 would like to submit a PR or a Design Doc for this?
Would love to get this in

[isker]

@tejal29 I will try to produce a PR for this. It looks to me that we could achieve all of the things you listed just by deleting the code that makes COPYs cacheable.

Like @kamaln7 said, we already calculate the composite cache key for non-cacheable COPYs:

kaniko/pkg/executor/build.go

Lines 186 to 187 in f20f495

	case *commands.CopyCommand:
	compositeKey = s.populateCopyCmdCompositeKey(command, v.From(), compositeKey)

This makes me wonder why COPYs were cacheable in the first place 🤔 .

[kamaln7]

Amazing, thanks @isker & @tejal29!