Image creation time inconsistent between builds

[Matthijsy]

Actual behavior
Currently running a kaniko build for the same Dockerfile twice (with caching enabled) returns a docker image with different value of Created in the manifest, even when nothing is changed. Using the --reproducible flag does not solve this, since that unsets this field, and we want to have the field.

Expected behavior
If all layers are exactly the same as the cache, the Created time of the destination Docker image is unchanged.

To Reproduce
Steps to reproduce the behavior:

1. Execute the kaniko executor twice with the following command:

/kaniko/executor
      --context "."
      --dockerfile "Dockerfile"
      --destination "${REGISTRY}/${IMAGE_NAME}"
      --cache-repo=${CACHE_REGISTRY}/cache-${IMAGE_NAME}
      --cache=true
      --cache-copy-layers=true
      --cache-run-layers=true

2. Pull both images: docker pull <registry>/<image>@sha256:<digest>
3. Check the Created field using docker inspect <registry>/<image>@sha256:<digest> | jq -r '.[0].Created'

Additional Information

• Dockerfile

FROM debian:10 AS test

RUN apt update
RUN apt install -y curl

RUN mkdir /test
WORKDIR /test

• Kaniko Image (fully qualified with digest)
  gcr.io/kaniko-project/executor@sha256:ac169723b2076f9d5804f4bc05c98397e286da6fdcdd5a09fdc179f06ccb3be1

• Kaniko output during first build

INFO[0000] Resolved base name debian:10 to test 
INFO[0000] Retrieving image manifest debian:10          
INFO[0000] Retrieving image debian:10 from registry index.docker.io 
INFO[0001] Retrieving image manifest debian:10          
INFO[0001] Returning cached image manifest              
INFO[0001] Built cross stage deps: map[]                
INFO[0001] Retrieving image manifest debian:10          
INFO[0001] Returning cached image manifest              
INFO[0001] Retrieving image manifest debian:10          
INFO[0001] Returning cached image manifest              
INFO[0001] Executing 0 build triggers                   
INFO[0001] Building stage 'debian:10' [idx: '0', base-idx: '-1'] 
INFO[0001] Checking for cached layer <cache_repo>/cache-test:b421f8ca3c1139f98a60204bda2a6976497e8cd1f53cb72009cbc35d0ce5d863... 
INFO[0001] No cached layer found for cmd RUN apt update 
INFO[0001] Unpacking rootfs as cmd RUN apt update requires it. 
INFO[0003] RUN apt update                               
INFO[0003] Initializing snapshotter ...                 
INFO[0003] Taking snapshot of full filesystem...        
INFO[0004] Cmd: /bin/sh                                 
INFO[0004] Args: [-c apt update]                        
INFO[0004] Running: [/bin/sh -c apt update]             
<apt output, removed for readability>
INFO[0014] Taking snapshot of full filesystem...        
INFO[0015] RUN mkdir /test                              
INFO[0015] Cmd: /bin/sh                                 
INFO[0015] Args: [-c mkdir /test]                       
INFO[0015] Running: [/bin/sh -c mkdir /test]            
INFO[0015] Taking snapshot of full filesystem...        
INFO[0015] Pushing layer <cache_registry>/cache-test:072a9673c8e0d578e15a8d874f53b571b7ac03e32243cacd4f8048f3ddf001d3 to cache now 
INFO[0015] Pushing image to <cache_registry>/cache-test:072a9673c8e0d578e15a8d874f53b571b7ac03e32243cacd4f8048f3ddf001d3 
INFO[0015] Pushed <cache_registry>/cache-test@sha256:0367e639324dc730d22c23160a005b5b9cb943568da6bacbb9c1c1c60ace017a 
INFO[0015] WORKDIR /test                                
INFO[0015] Cmd: workdir                                 
INFO[0015] Changed working directory to /test           
INFO[0015] Pushing layer <cache_registry>/cache-test:e449599e0237d86420f035de3adeb6fabb67ba43d54678268f1cbbca8ec19476 to cache now 
INFO[0015] Pushing image to <cache_registry>/cache-test:e449599e0237d86420f035de3adeb6fabb67ba43d54678268f1cbbca8ec19476 
INFO[0015] No files changed in this command, skipping snapshotting. 
INFO[0015] Pushed r<cache_registry>/cache-test@sha256:ff8682a468552deda470243726ca2d05bfce4e9d80c774644ce8232e8325a560 
INFO[0015] Pushing image to <registry>/test 
INFO[0018] Pushed <registry>/test@sha256:254bc1c9a37b190bc7ed7218357276bc715c40a574a627e0a386d8d3e92a029d 

• Kaniko output during second run

INFO[0000] Resolved base name debian:10 to test 
INFO[0000] Retrieving image manifest debian:10          
INFO[0000] Retrieving image debian:10 from registry index.docker.io 
INFO[0000] Retrieving image manifest debian:10          
INFO[0000] Returning cached image manifest              
INFO[0001] Built cross stage deps: map[]                
INFO[0001] Retrieving image manifest debian:10          
INFO[0001] Returning cached image manifest              
INFO[0001] Retrieving image manifest debian:10          
INFO[0001] Returning cached image manifest              
INFO[0001] Executing 0 build triggers                   
INFO[0001] Building stage 'debian:10' [idx: '0', base-idx: '-1'] 
INFO[0001] Checking for cached layer <cache_registry>/cache-test:b421f8ca3c1139f98a60204bda2a6976497e8cd1f53cb72009cbc35d0ce5d863... 
INFO[0001] Using caching version of cmd: RUN apt update 
INFO[0001] Checking for cached layer <cache_registry>/cache-test:072a9673c8e0d578e15a8d874f53b571b7ac03e32243cacd4f8048f3ddf001d3... 
INFO[0001] Using caching version of cmd: RUN apt install -y curl 
INFO[0001] Checking for cached layer <cache_registry>/cache-test:e449599e0237d86420f035de3adeb6fabb67ba43d54678268f1cbbca8ec19476... 
INFO[0001] Using caching version of cmd: RUN mkdir /test 
INFO[0001] Skipping unpacking as no commands require it. 
INFO[0001] RUN apt update                               
INFO[0001] Found cached layer, extracting to filesystem 
INFO[0002] RUN apt install -y curl                      
INFO[0002] Found cached layer, extracting to filesystem 
INFO[0002] RUN mkdir /test                              
INFO[0002] Found cached layer, extracting to filesystem 
INFO[0002] WORKDIR /test                                
INFO[0002] Cmd: workdir                                 
INFO[0002] Changed working directory to /test           
INFO[0002] No files changed in this command, skipping snapshotting. 
INFO[0002] Pushing image to <registry>/test 
INFO[0003] Pushed <registry>/test@sha256:7fbb2f29d23828650d8b7c22529aa5c6c1e0d40492b2bdc66309df10ff2b7391 

• Output of Created of first build

docker pull <registry>/test@sha256:254bc1c9a37b190bc7ed7218357276bc715c40a574a627e0a386d8d3e92a029d
docker inspect <registry>/test@sha256:254bc1c9a37b190bc7ed7218357276bc715c40a574a627e0a386d8d3e92a029d | jq -r '.[0].Created'
2022-12-19T13:26:58.355532708Z

• Output of Created of second build

docker pull <registry>/test@sha256:7fbb2f29d23828650d8b7c22529aa5c6c1e0d40492b2bdc66309df10ff2b7391
docker inspect <registry>/test@sha256:7fbb2f29d23828650d8b7c22529aa5c6c1e0d40492b2bdc66309df10ff2b7391 | jq -r '.[0].Created'
2022-12-19T13:27:32.67742304Z

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[wbolster]

maybe dup of #2005

[aaron-prindle]

@Matthijsy @wbolster can either of you confirm if this issue is still occuring with the latest release of kaniko? I believe the now fixed #2005 is a dupe of this

[wbolster]

i can confirm that the fix for #2005 fixed the reproducibility problems i experienced with earlier versions.

the build transcripts in this issue look like what i experienced, hence my earlier comment on this ticket.

so yeah, close it i'd say! 🚀

[Matthijsy]

Sorry for my late response! My issue however is not fixed. The #2005 is about using the --reproducible, which was not working correctly. However, I cannot use that flag, since that will set the 'Created' field in the metadata to unset. We want to use this field, but if two builds are exactly the same they should have the same 'Created' time.

@aaron-prindle Could you re-open this issue maybe? Of do you know a way that we can fix this behaviour?