BuildKit fails to use local images with Podman backend

[Trigus42]

Contributing guidelines and issue reporting guide

• I've read the contributing guidelines and wholeheartedly agree. I've also read the issue reporting guide.

Well-formed report checklist

• I have found a bug that the documentation does not mention anything about my problem
• I have found a bug that there are no open or closed issues that are related to my problem
• I have provided version/information about my environment and done my best to provide a reproducer

Description of bug

Bug description

When using Docker CLI with a Podman service backend (configured via DOCKER_HOST), BuildKit (the default builder) incorrectly attempts to pull local images from a remote registry (docker.io/library/...). This results in a build failure with a "connection refused" error.

This occurs when using image ids (e.g 7a3108859ba4, sha256:7a3108859ba4d240c9c3a68437e1d7d969af80913b9ee88a16c779ce713d7275) or tagged images (e.g. mylocalbaseimage, mylocalbaseimage:v1). When prepending localhost to the image tag (e.g. localhost/mylocalbaseimage:v1, buildkit attempts to pull the image from http://localhost/v2/mylocalbaseimage/manifests/v1

This bug might be related to #2204.

Reproduction

• Set DOCKER_HOST to point to the Podman socket
• Ensure a local image exists
• Use a Dockerfile with FROM mylocalimage
• Run docker build .

All commands described above

📦[user@dev test]$ echo $DOCKER_HOST
unix:////run/host/run/user/1000/podman/podman.sock
📦[user@dev test]$ docker build --load -f Dockerfile.base -t mylocalimage -t mylocalimage:v1 -t localhost/mylocalimage .
[+] Building 2.0s (7/7) FINISHED                                                                                                                                               docker-container:default
 => [internal] load build definition from Dockerfile.base                                                                                                                                          0.0s
 => => transferring dockerfile: 150B                                                                                                                                                               0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                                                                                                                   0.6s
 => [internal] load .dockerignore                                                                                                                                                                  0.0s
 => => transferring context: 2B                                                                                                                                                                    0.0s
 => [1/2] FROM docker.io/library/alpine:latest@sha256:a8560b36e8b8210634f77d9f7f9efd7ffa463e380b75e2e74aff4511df3ef88c                                                                             0.7s
 => => resolve docker.io/library/alpine:latest@sha256:a8560b36e8b8210634f77d9f7f9efd7ffa463e380b75e2e74aff4511df3ef88c                                                                             0.0s
 => => sha256:f18232174bc91741fdf3da96d85011092101a032a93a388b79e99e69c2d5c870 3.64MB / 3.64MB                                                                                                     0.6s
 => => extracting sha256:f18232174bc91741fdf3da96d85011092101a032a93a388b79e99e69c2d5c870                                                                                                          0.1s
 => [2/2] RUN echo "Echo from Dockerfile.base"                                                                                                                                                     0.2s
 => exporting to docker image format                                                                                                                                                               0.3s 
 => => exporting layers                                                                                                                                                                            0.1s
 => => exporting manifest sha256:b705dc630b60faad2c0c93874648f805ad0456e6313b3a6969e94fb8426fd077                                                                                                  0.0s
 => => exporting config sha256:c3200694bb316514f35083504963e047bed3a14b3081f913ec1ef802fb6d31d9                                                                                                    0.0s
 => => sending tarball                                                                                                                                                                             0.2s
 => importing to docker                                                                                                                                                                            0.0s
📦[user@dev test]$ docker image inspect mylocalimage
[
    {
        "Id": "sha256:c3200694bb316514f35083504963e047bed3a14b3081f913ec1ef802fb6d31d9",
        "RepoTags": [
            "localhost/mylocalimage:latest",
            "localhost/mylocalimage:v1"
        ],
        "RepoDigests": [
            "localhost/mylocalimage@sha256:77fc695ecf3f01ca9a984f1cc8cdc80c61f95d2600e1b688f7c8bbd7d587ce2e"
        ],
        "Parent": "",
        "Comment": "",
        "Created": "2025-05-24T21:14:42.782341968Z",
        "ContainerConfig": {
            "Hostname": "c3200694bb3",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": null,
            "Cmd": null,
            "Image": "",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": null
        },
        "DockerVersion": "",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "/bin/sh"
            ],
            "Image": "",
            "Volumes": null,
            "WorkingDir": "/",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": null
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 8123853,
        "VirtualSize": 8123853,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/home/user/.local/share/containers/storage/overlay/08000c18d16dadf9553d747a58cf44023423a9ab010aab96cf263d2216b8b350/diff",
                "UpperDir": "/home/user/.local/share/containers/storage/overlay/30656556fc7ff1d7e44b195a812ddb7221944d2d5be65ae9a5e712a96d780fca/diff",
                "WorkDir": "/home/user/.local/share/containers/storage/overlay/30656556fc7ff1d7e44b195a812ddb7221944d2d5be65ae9a5e712a96d780fca/work"
            },
            "Name": "overlay"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:08000c18d16dadf9553d747a58cf44023423a9ab010aab96cf263d2216b8b350",
                "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        },
        "Container": ""
    }
]
📦[user@dev test]$ docker run -it --rm mylocalimage whoami
root
📦[user@dev test]$ docker build -f Dockerfile.top --build-arg BASE_IMAGE=mylocalbaseimage .
[+] Building 1.1s (2/2) FINISHED                                                                                                                                               docker-container:default
 => [internal] load build definition from Dockerfile.top                                                                                                                                           0.0s
 => => transferring dockerfile: 170B                                                                                                                                                               0.0s
 => WARN: InvalidDefaultArgInFrom: Default value for ARG ${BASE_IMAGE} results in empty or invalid base image name (line 2)                                                                        0.0s
 => ERROR [internal] load metadata for docker.io/library/mylocalbaseimage:latest                                                                                                                   1.1s
------
 > [internal] load metadata for docker.io/library/mylocalbaseimage:latest:
------
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load

 1 warning found (use docker --debug to expand):
 - InvalidDefaultArgInFrom: Default value for ARG ${BASE_IMAGE} results in empty or invalid base image name (line 2)
Dockerfile.top:2
--------------------
   1 |     ARG BASE_IMAGE
   2 | >>> FROM ${BASE_IMAGE}
   3 |     
   4 |     RUN echo "Echo from Dockerfile.top"
--------------------
ERROR: failed to solve: mylocalbaseimage: failed to resolve source metadata for docker.io/library/mylocalbaseimage:latest: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
📦[user@dev test]$ docker build -f Dockerfile.top --build-arg BASE_IMAGE=mylocalbaseimage:v1 .
[+] Building 0.8s (2/2) FINISHED                                                                                                                                               docker-container:default
 => [internal] load build definition from Dockerfile.top                                                                                                                                           0.0s
 => => transferring dockerfile: 170B                                                                                                                                                               0.0s
 => WARN: InvalidDefaultArgInFrom: Default value for ARG ${BASE_IMAGE} results in empty or invalid base image name (line 2)                                                                        0.0s
 => ERROR [internal] load metadata for docker.io/library/mylocalbaseimage:v1                                                                                                                       0.7s
------
 > [internal] load metadata for docker.io/library/mylocalbaseimage:v1:
------
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load

 1 warning found (use docker --debug to expand):
 - InvalidDefaultArgInFrom: Default value for ARG ${BASE_IMAGE} results in empty or invalid base image name (line 2)
Dockerfile.top:2
--------------------
   1 |     ARG BASE_IMAGE
   2 | >>> FROM ${BASE_IMAGE}
   3 |     
   4 |     RUN echo "Echo from Dockerfile.top"
--------------------
ERROR: failed to solve: mylocalbaseimage:v1: failed to resolve source metadata for docker.io/library/mylocalbaseimage:v1: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
📦[user@dev test]$ docker build -f Dockerfile.top --build-arg BASE_IMAGE=localhost/mylocalbaseimage:v1 .
[+] Building 0.0s (2/2) FINISHED                                                                                                                                               docker-container:default
 => [internal] load build definition from Dockerfile.top                                                                                                                                           0.0s
 => => transferring dockerfile: 170B                                                                                                                                                               0.0s
 => WARN: InvalidDefaultArgInFrom: Default value for ARG ${BASE_IMAGE} results in empty or invalid base image name (line 2)                                                                        0.0s
 => ERROR [internal] load metadata for localhost/mylocalbaseimage:v1                                                                                                                               0.0s
------
 > [internal] load metadata for localhost/mylocalbaseimage:v1:
------
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load

 1 warning found (use docker --debug to expand):
 - InvalidDefaultArgInFrom: Default value for ARG ${BASE_IMAGE} results in empty or invalid base image name (line 2)
Dockerfile.top:2
--------------------
   1 |     ARG BASE_IMAGE
   2 | >>> FROM ${BASE_IMAGE}
   3 |     
   4 |     RUN echo "Echo from Dockerfile.top"
--------------------
ERROR: failed to solve: localhost/mylocalbaseimage:v1: failed to resolve source metadata for localhost/mylocalbaseimage:v1: failed to do request: Head "http://localhost/v2/mylocalbaseimage/manifests/v1": dial tcp [::1]:80: connect: connection refused
📦[user@dev test]$ docker build -f Dockerfile.top --build-arg BASE_IMAGE=sha256:c3200694bb316514f35083504963e047bed3a14b3081f913ec1ef802fb6d31d9 .
[+] Building 1.2s (2/2) FINISHED                                                                                                                                               docker-container:default
 => [internal] load build definition from Dockerfile.top                                                                                                                                           0.0s
 => => transferring dockerfile: 170B                                                                                                                                                               0.0s
 => WARN: InvalidDefaultArgInFrom: Default value for ARG ${BASE_IMAGE} results in empty or invalid base image name (line 2)                                                                        0.0s
 => ERROR [internal] load metadata for docker.io/library/sha256:c3200694bb316514f35083504963e047bed3a14b3081f913ec1ef802fb6d31d9                                                                   1.1s
------
 > [internal] load metadata for docker.io/library/sha256:c3200694bb316514f35083504963e047bed3a14b3081f913ec1ef802fb6d31d9:
------
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load

 1 warning found (use docker --debug to expand):
 - InvalidDefaultArgInFrom: Default value for ARG ${BASE_IMAGE} results in empty or invalid base image name (line 2)
Dockerfile.top:2
--------------------
   1 |     ARG BASE_IMAGE
   2 | >>> FROM ${BASE_IMAGE}
   3 |     
   4 |     RUN echo "Echo from Dockerfile.top"
--------------------
ERROR: failed to solve: sha256:c3200694bb316514f35083504963e047bed3a14b3081f913ec1ef802fb6d31d9: failed to resolve source metadata for docker.io/library/sha256:c3200694bb316514f35083504963e047bed3a14b3081f913ec1ef802fb6d31d9: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

Version information

$ podman -v
podman version 5.5.0

$ docker -v
Docker version 28.0.4, build b8034c0

[tonistiigi]

Podman does not support buildkit internally (afaik). What will happen is that buildx will detect that the docker socket has no native support and instead bootstrap a buildkit builder inside a container. You can see that from the "docker-container:" in your progress, and also if you would run docker buildx inspect. Container builder can't see things like docker (podman) image store outside of the container. Alternatives are to use a registry or --build-context pointing to OCI layout of the image you wish to access https://docs.docker.com/reference/cli/docker/buildx/build/#build-context, or of course to use actual Docker 😉 . You can also use docker buildx bake to chain builds from multiple Dockerfiles together without the need for an intermediate image.

[Trigus42]

Ah that makes sense. Thanks a lot!