[CI]: buildkit `Internal: not found`

[apostasie]

Description

This is happening regularly. Seen on experimental #3303 twice in the same run (rootless, if that matters).

Attached is a copy of the logs from tests relying on build.

Intuition (wild guess, actually) suggest this might be related to containerd delete / garbage-collection, and could possibly be a buildkitd issue.

Steps to reproduce the issue

Describe the results you received and expected

=== FAIL: cmd/nerdctl/container TestRunCopyingUpInitialContentsOnDockerfileVolume (0.62s)
    testutil.go:655: buildkitHost="unix:///run/user/1001/buildkit-nerdctl-test/buildkitd.sock"
    container_run_mount_linux_test.go:182: assertion failed: res.ExitCode is not exitCode: 
        Command:  /usr/local/bin/nerdctl --namespace=nerdctl-test build -t nerdctl-testruncopyingupinitialcontentsondockerfilevolume /tmp/TestRunCopyingUpInitialContentsOnDockerfileVolume3738922107/001
        ExitCode: 1
        Error:    exit status 1
        Stdout:   
        Stderr:   #1 [internal] load build definition from Dockerfile
        #1 transferring dockerfile: 188B done
        #1 DONE 0.0s
        
        #2 [internal] load metadata for ghcr.io/stargz-containers/alpine:3.13-org
        #2 DONE 0.1s
        
        #3 [internal] load .dockerignore
        #3 transferring context: 2B done
        #3 DONE 0.0s
        
        #4 [1/2] FROM ghcr.io/stargz-containers/alpine:3.13-org@sha256:ec14c7992a97fc11425907e908340c6c3d6ff602f5f13d899e6b7027c9b4133a
        #4 resolve ghcr.io/stargz-containers/alpine:3.13-org@sha256:ec14c7992a97fc11425907e908340c6c3d6ff602f5f13d899e6b7027c9b4133a 0.0s done
        #4 CACHED
        
        #5 [2/2] RUN mkdir -p /mnt && echo hi > /mnt/initial_file
        #5 DONE 0.2s
        
        #6 exporting to image
        #6 exporting layers 0.0s done
        #6 exporting manifest sha256:c54c31e481be28beb9b5893fd6569dc20c123705ea4246aa49ef503e993c96f9 done
        #6 exporting config sha256:b60e26ee3e57f3fb47e4bf798dd06502add47433c3d668007a0d8d14d11502ae done
        #6 naming to docker.io/library/nerdctl-testruncopyingupinitialcontentsondockerfilevolume:latest done
        #6 unpacking to docker.io/library/nerdctl-testruncopyingupinitialcontentsondockerfilevolume:latest 0.0s done
        #6 DONE 0.1s
        error: failed to solve: Internal: not found
        
        

=== FAIL: cmd/nerdctl/container TestRunCopyingUpInitialContentsOnVolume (0.48s)
    testutil.go:655: buildkitHost="unix:///run/user/1001/buildkit-nerdctl-test/buildkitd.sock"
    container_run_mount_linux_test.go:154: assertion failed: res.ExitCode is not exitCode: 
        Command:  /usr/local/bin/nerdctl --namespace=nerdctl-test build -t nerdctl-testruncopyingupinitialcontentsonvolume /tmp/TestRunCopyingUpInitialContentsOnVolume1393391841/001
        ExitCode: 1
        Error:    exit status 1
        Stdout:   
        Stderr:   #1 [1/2] FROM ghcr.io/stargz-containers/alpine:3.13-org@sha256:ec14c7992a97fc11425907e908340c6c3d6ff602f5f13d899e6b7027c9b4133a
        #1 resolve ghcr.io/stargz-containers/alpine:3.13-org@sha256:ec14c7992a97fc11425907e908340c6c3d6ff602f5f13d899e6b7027c9b4133a 0.0s done
        #1 CACHED
        
        #2 [internal] load build definition from Dockerfile
        #2 transferring dockerfile: 176B done
        #2 DONE 0.0s
        
        #3 [2/2] RUN mkdir -p /mnt && echo hi > /mnt/initial_file
        #3 DONE 0.2s
        
        #4 [internal] load metadata for ghcr.io/stargz-containers/alpine:3.13-org
        #4 DONE 0.1s
        
        #5 [internal] load .dockerignore
        #5 transferring context: 2B done
        #5 DONE 0.0s
        
        #6 exporting to image
        #6 exporting layers done
        #6 exporting manifest sha256:20fe0bd0c4692c355a83232ef93987b9a638cefe724ccb7593d21897e3cdd437 done
        #6 exporting config sha256:478c4aac618d55bb333a7c16c8c0c6d69db6958fee358d562ffedd1232e6719c done
        #6 naming to docker.io/library/nerdctl-testruncopyingupinitialcontentsonvolume:latest done
        #6 unpacking to docker.io/library/nerdctl-testruncopyingupinitialcontentsonvolume:latest 0.0s done
        #6 DONE 0.0s
        error: failed to solve: Internal: not found

What version of nerdctl are you using?

main

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

No response

[apostasie]

@AkihiroSuda this is presumably third-party