Problems with --reproducible (modified timestamp)

[leongross]

Actual behavior
Using a minimal Dockerfile and that add an empty file and flag --reproducible results in different build hashes when built multiple times.

Expected behavior
The same hash for the docker images.

To Reproduce
Steps to reproduce the behavior:

1. Create Dockerfile

FROM ubuntu@sha256:7cfe75438fc77c9d7235ae502bf229b15ca86647ac01c844b272b56326d56184
COPY empty /

2. Create empty file

touch empty

3. Run kaniko build

docker run \
    -v "$(pwd)"/context/:/workspace \
    -v "$(pwd)"/config.json:/kaniko/.docker/config.json:ro \
    gcr.io/kaniko-project/executor \
    --reproducible \
    --dockerfile Dockerfile \
    --context dir:///workspace/ \
    --destination <registry> \
    --cache=false

Additional Information

• Dockerfile: above
  Please provide either the Dockerfile you're trying to build or one that can reproduce this error.
• Build Context

$ tree
── config.json
├── context
│   ├── Dockerfile
│   └── empty
└── run.sh (containing the command)

• Kaniko Image: gcr.io/kaniko-project/executor latest 14c90714063c 4 weeks ago 63.1MB

EDIT 1:
Running container-diff does not yield any differences

$ container-diff diff <IMG_0> <IMG_1> --type=history --type=file --type=size                                                                                                                                                                   

-----File-----

These entries have been added to <IMG_0> None

These entries have been deleted from <IMG_0> None

These entries have been changed between <IMG_0> and <IMG_1>: None

-----History-----

Docker history lines found only in <IMG_0>: None

Docker history lines found only in <IMG_1>: None

-----Size-----

Image size difference between <IMG_0> and <IMG_1>: None

Triage Notes for the Maintainers

EDIT 2:
I used diffoscope to inspect the file system layers of the docker containers and found the following differences:

$ sudo diffoscope /var/lib/docker/overlay2/b961fc9f522d3e9bd9ab215e1292e5a6a5f461e61913bf47f041d49d7e3c3a07 /var/lib/docker/overlay2/fefa77080b6ef9f2bea5a425761f94d0f955c10768945f1626d211aba298ee34/
+++ /var/lib/docker/overlay2/fefa77080b6ef9f2bea5a425761f94d0f955c10768945f1626d211aba298ee34/
├── stat {}
│ @@ -1,8 +1,8 @@
│  
│    Size: 4096      	Blocks: 8          IO Block: 4096   directory
│  Device: 259,3	Links: 4
│  Access: (0710/drwx--x---)  Uid: (    0/    root)   Gid: (    0/    root)
│  
│ -Modify: 2022-10-31 11:25:57.856492186 +0000
│ +Modify: 2022-10-31 11:29:02.389132829 +0000
│   --- /var/lib/docker/overlay2/b961fc9f522d3e9bd9ab215e1292e5a6a5f461e61913bf47f041d49d7e3c3a07/committed
├── +++ /var/lib/docker/overlay2/fefa77080b6ef9f2bea5a425761f94d0f955c10768945f1626d211aba298ee34/committed
│ ├── stat {}
│ │ @@ -1,8 +1,8 @@
│ │  
│ │    Size: 0         	Blocks: 0          IO Block: 4096   regular empty file
│ │  Device: 259,3	Links: 1
│ │  Access: (0600/-rw-------)  Uid: (    0/    root)   Gid: (    0/    root)
│ │  
│ │ -Modify: 2022-10-31 11:25:57.856492186 +0000
│ │ +Modify: 2022-10-31 11:41:02.903334450 +0000
│   --- /var/lib/docker/overlay2/b961fc9f522d3e9bd9ab215e1292e5a6a5f461e61913bf47f041d49d7e3c3a07/diff
├── +++ /var/lib/docker/overlay2/fefa77080b6ef9f2bea5a425761f94d0f955c10768945f1626d211aba298ee34/diff
│ ├── stat {}
│ │ @@ -1,8 +1,8 @@
│ │  
│ │    Size: 4096      	Blocks: 8          IO Block: 4096   directory
│ │  Device: 259,3	Links: 2
│ │  Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
│ │  
│ │ +Modify: 2022-10-31 11:25:51.046396023 +0000
│ │ -Modify: 2022-10-31 11:25:57.829825142 +0000
│ ├── stat {}
│ │ @@ -1,8 +1,8 @@
│ │  
│ │    Size: 4096      	Blocks: 8          IO Block: 4096   directory
│ │  Device: 259,3	Links: 2
│ │  Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
│ │  
│ │ -Modify: 2022-10-31 11:25:57.829825142 +0000
│ │ +Modify: 2022-10-31 11:25:51.046396023 +0000
│   --- /var/lib/docker/overlay2/b961fc9f522d3e9bd9ab215e1292e5a6a5f461e61913bf47f041d49d7e3c3a07/link
├── +++ /var/lib/docker/overlay2/fefa77080b6ef9f2bea5a425761f94d0f955c10768945f1626d211aba298ee34/link
│ @@ -1 +1 @@
│ -MJTPWNJJMXPGAGOR4AU5SGV3EU
│ +YY2LS2LHD5FII6IXAHIGIUHWCT
│ ├── stat {}
│ │ @@ -1,8 +1,8 @@
│ │  
│ │    Size: 26        	Blocks: 8          IO Block: 4096   regular file
│ │  Device: 259,3	Links: 1
│ │  Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
│ │  
│ │ -Modify: 2022-10-31 11:25:57.806491480 +0000
│ │ +Modify: 2022-10-31 11:25:51.023062360 +0000
│   --- /var/lib/docker/overlay2/b961fc9f522d3e9bd9ab215e1292e5a6a5f461e61913bf47f041d49d7e3c3a07/lower
├── +++ /var/lib/docker/overlay2/fefa77080b6ef9f2bea5a425761f94d0f955c10768945f1626d211aba298ee34/lower
│ ├── stat {}
│ │ @@ -1,8 +1,8 @@
│ │  
│ │    Size: 57        	Blocks: 8          IO Block: 4096   regular file
│ │  Device: 259,3	Links: 1
│ │  Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
│ │  
│ │ -Modify: 2022-10-31 11:25:57.806491480 +0000
│ │ +Modify: 2022-10-31 11:25:51.023062360 +0000
│   --- /var/lib/docker/overlay2/b961fc9f522d3e9bd9ab215e1292e5a6a5f461e61913bf47f041d49d7e3c3a07/work
├── +++ /var/lib/docker/overlay2/fefa77080b6ef9f2bea5a425761f94d0f955c10768945f1626d211aba298ee34/work
│ ├── stat {}
│ │ @@ -1,8 +1,8 @@
│ │  
│ │    Size: 4096      	Blocks: 8          IO Block: 4096   directory
│ │  Device: 259,3	Links: 3
│ │  Access: (0700/drwx------)  Uid: (    0/    root)   Gid: (    0/    root)
│ │  
│ │ +Modify: 2022-10-31 11:25:51.023062360 +0000
│ │ -Modify: 2022-10-31 11:25:57.806491480 +0000
│ │   --- /var/lib/docker/overlay2/b961fc9f522d3e9bd9ab215e1292e5a6a5f461e61913bf47f041d49d7e3c3a07/work/work
│ ├── +++ /var/lib/docker/overlay2/fefa77080b6ef9f2bea5a425761f94d0f955c10768945f1626d211aba298ee34/work/work
│ │ ├── stat {}
│ │ │ @@ -1,8 +1,8 @@
│ │ │  
│ │ │    Size: 4096      	Blocks: 8          IO Block: 4096   directory
│ │ │  Device: 259,3	Links: 2
│ │ │  Access: (0000/d---------)  Uid: (    0/    root)   Gid: (    0/    root)
│ │ │  
│ │ │ -Modify: 2022-10-31 11:25:57.806491480 +0000
│ │ │ +Modify: 2022-10-31 11:25:51.023062360 +0000
│ │ ├── stat {}
│ │ │ @@ -1,8 +1,8 @@
│ │ │  
│ │ │    Size: 4096      	Blocks: 8          IO Block: 4096   directory
│ │ │  Device: 259,3	Links: 2
│ │ │  Access: (0000/d---------)  Uid: (    0/    root)   Gid: (    0/    root)
│ │ │  
│ │ │ +Modify: 2022-10-31 11:25:51.023062360 +0000
│ │ │ -Modify: 2022-10-31 11:25:57.806491480 +0000
│ ├── stat {}
│ │ @@ -1,8 +1,8 @@
│ │  
│ │    Size: 4096      	Blocks: 8          IO Block: 4096   directory
│ │  Device: 259,3	Links: 3
│ │  Access: (0700/drwx------)  Uid: (    0/    root)   Gid: (    0/    root)
│ │  
│ │ -Modify: 2022-10-31 11:25:57.806491480 +0000
│ │ +Modify: 2022-10-31 11:25:51.023062360 +0000

It looks like they only differ in the modification time, which I guess should not happen when the --reproducible flag is passed, right?

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[Sjd-Risca]

Unlucky this is an already known issue, since version 1.8 as described here by bug 2005. Into such issue is described also the reason behind the anomalous behavior.