WCOW: test fails with file used by another process

[tonistiigi]

https://github.com/moby/buildkit/actions/runs/13639852250/job/38127254409?pr=5806#step:8:47

    dockerfile_test.go:1508: 
        	Error Trace:	D:/a/buildkit/buildkit/frontend/dockerfile/dockerfile_test.go:1508
        	            				D:/a/buildkit/buildkit/util/testutil/integration/run.go:103
        	            				D:/a/buildkit/buildkit/util/testutil/integration/run.go:240
        	Error:      	Received unexpected error:
        	            	failed to compute cache key: failed to calculate checksum of ref swoah7anzxal82qqgw104woil::90nr5607yu8c0st4nmx1p5na9: failed to mount {windows-layer C:\Users\RUNNER~1\AppData\Local\Temp\bktest_containerd2597383045\root\io.containerd.snapshotter.v1.windows\snapshots\4  [parentLayerPaths=["C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\bktest_containerd2597383045\\root\\io.containerd.snapshotter.v1.windows\\snapshots\\3"] ro]}: failed to activate layer C:\Users\RUNNER~1\AppData\Local\Temp\bktest_containerd2597383045\root\io.containerd.snapshotter.v1.windows\snapshots\4: hcsshim::ActivateLayer failed in Win32: The process cannot access the file because it is being used by another process. (0x20): failed to activate layer C:\Users\RUNNER~1\AppData\Local\Temp\bktest_containerd2597383045\root\io.containerd.snapshotter.v1.windows\snapshots\4: hcsshim::ActivateLayer failed in Win32: The process cannot access the file because it is being used by another process. (0x20)
        	            	github.com/moby/buildkit/util/stack.Enable
        	            		D:/a/buildkit/buildkit/util/stack/stack.go:82
        	            	github.com/moby/buildkit/util/grpcerrors.FromGRPC
        	            		D:/a/buildkit/buildkit/util/grpcerrors/grpcerrors.go:204
        	            	github.com/moby/buildkit/util/grpcerrors.UnaryClientInterceptor
        	            		D:/a/buildkit/buildkit/util/grpcerrors/intercept.go:41
        	            	google.golang.org/grpc.(*ClientConn).Invoke

@profnandaa

[tonistiigi]

https://github.com/moby/buildkit/actions/runs/13704980284/job/38328092646?pr=5818#step:8:137

    sandbox.go:205: time="2025-03-06T17:58:20Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = failed to compute cache key: failed to calculate checksum of ref 4lv73nm9orh3ozl3fiwt3mxjf::ermuurq85748nfcfjlgp4bf62: failed to mount {windows-layer C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\bktest_containerd1553088924\\root\\io.containerd.snapshotter.v1.windows\\snapshots\\4  [parentLayerPaths=[\"C:\\\\Users\\\\RUNNER~1\\\\AppData\\\\Local\\\\Temp\\\\bktest_containerd1553088924\\\\root\\\\io.containerd.snapshotter.v1.windows\\\\snapshots\\\\3\"] ro]}: failed to activate layer C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\bktest_containerd1553088924\\root\\io.containerd.snapshotter.v1.windows\\snapshots\\4: hcsshim::ActivateLayer failed in Win32: The process cannot access the file because it is being used by another process. (0x20): failed to activate layer C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\bktest_containerd1553088924\\root\\io.containerd.snapshotter.v1.windows\\snapshots\\4: hcsshim::ActivateLayer failed in Win32: The process cannot access the file because it is being used by another process. (0x20)" spanID=d4fc2b40826ac0bc traceID=9cfb901e2e595cea4549db684385709a
    sandbox.go:205: failed to compute cache key: failed to calculate checksum of ref 4lv73nm9orh3ozl3fiwt3mxjf::ermuurq85748nfcfjlgp4bf62: failed to mount {windows-layer C:\Users\RUNNER~1\AppData\Local\Temp\bktest_containerd1553088924\root\io.containerd.snapshotter.v1.windows\snapshots\4  [parentLayerPaths=["C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\bktest_containerd1553088924\\root\\io.containerd.snapshotter.v1.windows\\snapshots\\3"] ro]}: failed to activate layer C:\Users\RUNNER~1\AppData\Local\Temp\bktest_containerd1553088924\root\io.containerd.snapshotter.v1.windows\snapshots\4: hcsshim::ActivateLayer failed in Win32: The process cannot access the file because it is being used by another process. (0x20): failed to activate layer C:\Users\RUNNER~1\AppData\Local\Temp\bktest_containerd1553088924\root\io.containerd.snapshotter.v1.windows\snapshots\4: hcsshim::ActivateLayer failed in Win32: The process cannot access the file because it is being used by another process. (0x20)
    sandbox.go:205: 3360 33a2fb0 buildkitd --containerd-worker-gc=false --containerd-worker=true --containerd-worker-addr npipe:////./pipe/containerd-bktest_containerd1553088924 --containerd-worker-labels=org.mobyproject.buildkit.worker.sandbox=true --config=C:\Users\RUNNER~1\AppData\Local\Temp\bktest_config1976606386\buildkitd.toml --root C:\Users\RUNNER~1\AppData\Local\Temp\bktest_buildkitd1272720178 --addr npipe:////./pipe/buildkitd-bktest_buildkitd1272720178 --debug
    sandbox.go:205: github.com/moby/buildkit/snapshot.(*localMounter).Mount

[profnandaa]

I have to been able to finally repro this outside the test scenario. Here is a basic script that I have fine-tuned for this, mostly you will just run it once to hit the issue, sometimes twice.

# .\repro.ps1
param(
	[Parameter(Position = 0)]
	[int]$Count = 50,
	[Parameter(Position = 1)]
	[int]$Pause = 3
)

$dockerfile = @"
FROM mcr.microsoft.com/windows/nanoserver:ltsc2022 AS build
RUN mkdir out\sub && mklink /D sub out\sub && mklink /D sub2 out\sub && echo data> sub\foo 

FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
COPY --from=build /sub/foo .
COPY --from=build /sub2/foo bar
"@

if (-not (Test-Path -Path ".\Dockerfile")) {
	Set-Content -Path .\Dockerfile -Value $dockerfile
}

$cmd = "buildctl build --frontend dockerfile.v0 --local context=. --local dockerfile=. --output type=image,name=docker.io/profnandaa/repro-5807,push=false --progress plain --no-cache"

for ($i = 1; $i -le $Count; $i++) { 
	Write-Host "`n=== Run $i`n"
	iex $cmd
	sleep $Pause
}

For instance, got the error on run 15:

<snip>
#7 [stage-1 3/3] COPY --from=build /sub2/foo bar
#7 ERROR: failed to calculate checksum of ref p11yciubfsq2s4670o7j7htlv::q0plonkxfhn1jbzipauld7bn2: failed to mount {windows-layer C:\ProgramData\containerd\root\io.containerd.snapshotter.v1.windows\snapshots\6247  [parentLayerPaths=["C:\\ProgramData\\containerd\\root\\io.containerd.snapshotter.v1.windows\\snapshots\\1891"] ro]}: failed to activate layer C:\ProgramData\containerd\root\io.containerd.snapshotter.v1.windows\snapshots\6247: hcsshim::ActivateLayer failed in Win32: The process cannot access the file because it is being used by another process. (0x20): failed to activate layer C:\ProgramData\containerd\root\io.containerd.snapshotter.v1.windows\snapshots\6247: hcsshim::ActivateLayer failed in Win32: The process cannot access the file because it is being used by another process. (0x20)
------
 > [stage-1 3/3] COPY --from=build /sub2/foo bar:
------
Dockerfile:6
--------------------
   4 |     FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
   5 |     COPY --from=build /sub/foo .
   6 | >>> COPY --from=build /sub2/foo bar
   7 |
--------------------
error: failed to solve: failed to compute cache key: failed to calculate checksum of ref p11yciubfsq2s4670o7j7htlv::q0plonkxfhn1jbzipauld7bn2: failed to mount {windows-layer C:\ProgramData\containerd\root\io.containerd.snapshotter.v1.windows\snapshots\6247  [parentLayerPaths=["C:\\ProgramData\\containerd\\root\\io.containerd.snapshotter.v1.windows\\snapshots\\1891"] ro]}: failed to activate layer C:\ProgramData\containerd\root\io.containerd.snapshotter.v1.windows\snapshots\6247: hcsshim::ActivateLayer failed in Win32: The process cannot access the file because it is being used by another process. (0x20): failed to activate layer C:\ProgramData\containerd\root\io.containerd.snapshotter.v1.windows\snapshots\6247: hcsshim::ActivateLayer failed in Win32: The process cannot access the file because it is being used by another process. (0x20)

=== Run 16
<snip>

PS. Without the pauses in between, you can even run 100x without hitting the error. With longer pauses too. Somehow 3 seconds seems to be the most successful.

[profnandaa]

After capturing ProcMon logs, I find 3 other processes (other than buildkitd) that are accessing the particular C:\\ProgramData\\containerd\\root\\io.containerd.snapshotter.v1.windows\\snapshots\\XXXX\\sandbox.vhdx around the same time I hit the error: containerd.exe, MsSense.exe(Defender), containerd-shim-runhcs-v1.exe.

I had placed a breakpoint at the place:

> [Breakpoint 11] github.com/moby/buildkit/snapshot.(*localMounter).Mount() C:/dev/core-containers/buildkit/snapshot/localmounter_windows.go:53 (hits goroutine(198264):1 total:1) (PC: 0x10aed85)
Warning: listing may not match stale executable
    48:                 if err := bindfilter.ApplyFileBinding(dir, m.Source, m.ReadOnly()); err != nil {
    49:                         return "", errors.Wrapf(err, "failed to mount %v", m)
    50:                 }
    51:         } else {
    52:                 if err := m.Mount(dir); err != nil {
=>  53:                         return "", errors.Wrapf(err, "failed to mount %v", m)
    54:                 }
    55:         }
    56:
    57:         lm.target = dir
    58:         return lm.target, nil
(dlv) p err
error(*fmt.wrapError) *{
        msg: "failed to activate layer C:\\ProgramData\\containerd\\root\\io.containerd.snapshotter.v1.windows\\snapshots\\4894: hcsshim::ActivateLayer failed in Win32: The process cannot access the file because it is being used by another process. (0x20)",
        err: error(*github.com/Microsoft/hcsshim/internal/hcserror.HcsError) *{
                title: "hcsshim::ActivateLayer",
                rest: "",
                Err: error(syscall.Errno) *(*error)(0xc0008d2770),},}

@TBBle -- there's some time back you looked into something close to this (though not related) - containerd/containerd#5133, might you have some tips on how I can go about:
(1) getting to pin-point the other process in contention of the file.
(2) possible fixes -- I currently have just a hack here where I'm retrying once on failure (works).