FROM/layer-extraction on ltsc2019 fails: link operation for `Windows/INF/basicrender.inf` on cross-platform building from Linux

[markmandel]

Describe the bug

COPY commands have been working normally for years with our lts2019 windows container build on Agones, but started failing today with an error message of:

Dockerfile.windows:18
--------------------
  16 |     FROM mcr.microsoft.com/windows/servercore:${WINDOWS_VERSION}
  17 |
  18 | >>> COPY ./bin/sdk-server.windows.amd64.exe /agones/sdk-server.exe
  19 |     COPY ./bin/LICENSES ./bin/dependencies-src.tgz /agones/
  20 |
--------------------
ERROR: failed to solve: failed to compute cache key: mount callback failed on /tmp/containerd-mount3356703247: link /tmp/containerd-mount3356703247/Windows/INF/basicrender.inf /tmp/containerd-mount3356703247/Windows/System32/DriverStore/FileRepository/basicrender.inf_amd64_efdc64af60c69a6d/basicrender.inf: no such file or directory
make: *** [Makefile:619: build-agones-sdk-image-windows-ltsc2019] Error 1

Where ${WINDOWS_VERSION} is ltsc2019

To Reproduce

To reproduce this, just try copying something in:

ARG WINDOWS_VERSION=ltsc2019
FROM mcr.microsoft.com/windows/servercore:${WINDOWS_VERSION}

COPY ./emptyfile /emptyfile

(Assuming you have a buildx builder already)

❯ touch emptyfile
❯ docker buildx build --platform windows/amd64 --builder windows-builder-ltsc2019 --tag=windows-test .
[+] Building 18.6s (6/6) FINISHED                                                                                                                                                                                                                                                                                                              docker-container:windows-builder-ltsc2019
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                                                                                                                0.1s
 => => transferring dockerfile: 156B                                                                                                                                                                                                                                                                                                                                                0.0s
 => [internal] load metadata for mcr.microsoft.com/windows/servercore:ltsc2019                                                                                                                                                                                                                                                                                                      0.1s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                                                                                   0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                                                                                                                     0.0s
 => [internal] load build context                                                                                                                                                                                                                                                                                                                                                   0.0s
 => => transferring context: 28B                                                                                                                                                                                                                                                                                                                                                    0.0s
 => [1/2] FROM mcr.microsoft.com/windows/servercore:ltsc2019@sha256:3c97a5c1c32ddb346c190f00a588da6e682a9a8160869f4969edfd7c6e4d1c03                                                                                                                                                                                                                                               18.3s
 => => resolve mcr.microsoft.com/windows/servercore:ltsc2019@sha256:3c97a5c1c32ddb346c190f00a588da6e682a9a8160869f4969edfd7c6e4d1c03                                                                                                                                                                                                                                                0.0s
 => => extracting sha256:0dd0445527a5079720e935502b31de927b8e22e5ca358026cf0bc8845c5ba5ce                                                                                                                                                                                                                                                                                          18.3s
 => ERROR [2/2] COPY ./emptyfile /emptyfile                                                                                                                                                                                                                                                                                                                                         0.0s
------
 > [2/2] COPY ./emptyfile /emptyfile:
------
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
Dockerfile:4
--------------------
   2 |     FROM mcr.microsoft.com/windows/servercore:${WINDOWS_VERSION}
   3 |
   4 | >>> COPY ./emptyfile /emptyfile
   5 |
--------------------
ERROR: failed to solve: failed to compute cache key: mount callback failed on /tmp/containerd-mount1556330329: link /tmp/containerd-mount1556330329/Windows/INF/basicrender.inf /tmp/containerd-mount1556330329/Windows/System32/DriverStore/FileRepository/basicrender.inf_amd64_efdc64af60c69a6d/basicrender.inf: no such file or directory

Expected behavior

The image should build 😃

Configuration:

• Edition: ltsc2019
• Base Image being used: mcr.microsoft.com/windows/servercore
• Container engine: docker, Running on Linux (debian).
• Container Engine version both 24.x and 26.x

Additional context

You can see a full build output from the Agones build pipeline here: https://console.cloud.google.com/cloud-build/builds/70b984e2-132b-4d1a-915a-862cb03f4830;step=16?e=13803378&project=agones-images

Using the previous SHA of @sha256:6fdf140282a2f809dae9b13fe441635867f0a27c33a438771673b8da8f3348a4 worked.

[markmandel]

Also possibly worth noting ltsc2022, works still. We tested it!

[jsturtevant]

which image is Using the previous SHA of @sha256:6fdf140282a2f809dae9b13fe441635867f0a27c33a438771673b8da8f3348a4 worked. in reference too?

[markmandel]

which image is Using the previous SHA of @sha256:6fdf140282a2f809dae9b13fe441635867f0a27c33a438771673b8da8f3348a4 worked. in reference too?

mcr.microsoft.com/windows/servercore:ltsc2019

[markmandel]

For reference, we have this PR now in place to unblock CI: googleforgames/agones#3829

[claudiubelu]

Tried it myself on a build node I have been using for years, though I have updated docker buildx since:

docker buildx version
github.com/docker/buildx v0.12.1 30feaa1a915b869ebc2eea6328624b49facd4bfb

I did use this version before KubeCon, and I did use mcr.microsoft.com/windows/servercore:ltsc2019 as a base image for the presentation where I talked about building Windows images. So, I think the image is the issue (Microsoft publishes a new image monthly).

[jsturtevant]

This looks to be an issue with the image patches released Yesterday (May 14th). Work around is use April's patch images as done in #493 (comment)

[jsturtevant]

/cc @akarshm

[jsturtevant]

Adding link to the slack discussion where we narrowed it down to the patch release https://kubernetes.slack.com/archives/C0SJ4AFB7/p1715733067064539

[jsturtevant]

/cc @profnandaa

[jsafrane]

I might be doing something wrong, but 41f42aa4ad39d85e4d30642b8111ca6454ca2275f188f012934b9afbaf63a647 does not work for me.

Dockerfile.reproduce:

FROM mcr.microsoft.com/windows/servercore:ltsc2019@sha256:41f42aa4ad39d85e4d30642b8111ca6454ca2275f188f012934b9afbaf63a647
RUN echo hello world > file

Build output:

docker build -f Dockerfile.reproduce  --platform=windows/amd64 .
[+] Building 458.1s (4/5)                                                                                                                                                                                                                                                                                      docker:default
 => [internal] load build definition from Dockerfile.reproduce                                                                                                                                                                                                                                                           0.0s
 => => transferring dockerfile: 202B                                                                                                                                                                                                                                                                                     0.0s
 => [internal] load metadata for mcr.microsoft.com/windows/servercore:ltsc2019@sha256:41f42aa4ad39d85e4d30642b8111ca6454ca2275f188f012934b9afbaf63a647                                                                                                                                                                   0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                        0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                                                          0.0s
 => ERROR [1/2] FROM mcr.microsoft.com/windows/servercore:ltsc2019@sha256:41f42aa4ad39d85e4d30642b8111ca6454ca2275f188f012934b9afbaf63a647                                                                                                                                                                             457.9s
 => => resolve mcr.microsoft.com/windows/servercore:ltsc2019@sha256:41f42aa4ad39d85e4d30642b8111ca6454ca2275f188f012934b9afbaf63a647                                                                                                                                                                                     0.0s
 => => sha256:41f42aa4ad39d85e4d30642b8111ca6454ca2275f188f012934b9afbaf63a647 375B / 375B                                                                                                                                                                                                                               0.0s
 => => sha256:f44fdada7eb2e71f80c964b2759421f4f7645a2df97a12aa82f66a8d50410414 596B / 596B                                                                                                                                                                                                                               0.0s
 => => sha256:b51a50ce03f12744163cde05017bec7149d704963fe41aed4b4a78472a98a3f8 788B / 788B                                                                                                                                                                                                                               0.0s
 => => sha256:cb524f6f22159378ea820d234d80ca09b79c2f0cc91315eeef11904e3ff36a21 1.60GB / 1.60GB                                                                                                                                                                                                                         410.6s
 => => sha256:9a0dd2e08eec1795898cdc8ce6217e961b67cb085e11998e2aa5e462cf68eb4e 573.97MB / 573.97MB                                                                                                                                                                                                                     219.4s
 => => extracting sha256:cb524f6f22159378ea820d234d80ca09b79c2f0cc91315eeef11904e3ff36a21                                                                                                                                                                                                                               38.1s
 => => extracting sha256:9a0dd2e08eec1795898cdc8ce6217e961b67cb085e11998e2aa5e462cf68eb4e                                                                                                                                                                                                                                8.5s
------
 > [1/2] FROM mcr.microsoft.com/windows/servercore:ltsc2019@sha256:41f42aa4ad39d85e4d30642b8111ca6454ca2275f188f012934b9afbaf63a647:
------
WARNING: current commit information was not captured by the build: git was not found in the system: exec: "git": executable file not found in $PATH
Dockerfile.reproduce:1
--------------------
   1 | >>> FROM mcr.microsoft.com/windows/servercore:ltsc2019@sha256:41f42aa4ad39d85e4d30642b8111ca6454ca2275f188f012934b9afbaf63a647
   2 |     
   3 |     
--------------------
ERROR: failed to solve: failed to register layer: link /Files/Program Files/common files/Microsoft Shared/Ink/en-US/micaut.dll.mui /Files/Program Files (x86)/common files/Microsoft Shared/ink/en-US/micaut.dll.mui: no such file or directory

Fedora 39, with docker-ce-27.0.3-1.fc39.x86_64.

I tried plain mcr.microsoft.com/windows/servercore:ltsc2019, the same result (and the same SHA).

[lemeurherve]

@jsafrane I got the same error as you when trying building Windows images with docker build on MacOS.

Try using docker buildx instead:

docker buildx build -f Dockerfile.reproduce --platform windows/amd64 .

[jsafrane]

docker buildx build -f Dockerfile.reproduce --platform windows/amd64 .

Same error as docker build.

[lemeurherve]

I should have looked more closely your Dockerfile.reproduce: you can't use RUN instructions if you want to build Windows images on another host than Windows.
(See for example https://stackoverflow.com/a/71910784/4074148)

Example of a working Windows Dockerfile built on MacOS:

FROM mcr.microsoft.com/windows/servercore:ltsc2019@sha256:41f42aa4ad39d85e4d30642b8111ca6454ca2275f188f012934b9afbaf63a647
COPY README.md README.md

[jsafrane]

Docker did not even reach RUN, it's stuck at unpacking layers from FROM. It fails with COPY too.

[profnandaa]

sorry about that, reopening to investigate.

[jsafrane]

It must be some issue in my setup. I can see that Kubernetes is able to build Windows images on Linux based on 41f42aa4ad39d85e4d30642b8111ca6454ca2275f188f012934b9afbaf63a647 and that's what matters to me (logs, search for the SHA there). It was failing with the old image.

/close

[akarshm]

Thanks, @jsafrane. Closing this issue.