Docker "--platform" broken for native sub-architectures

[satmandu]

Description

Running an armv7l container on aarch64 host fails to show armv7l used.
Similarly running an i386 container on x86_64 host fails to show i386 used.

Steps to reproduce the issue:

1. On a x86_64 host
   Use this Dockerfile:

FROM debian:buster
CMD uname -m

2. docker buildx build --platform linux/386 -t 386-build . --load
3. docker run --platform linux/386 --rm 386-build

Describe the results you received:

x86_64

Describe the results you expected:

i686

Output of docker version:

Client: Docker Engine - Community
 Version:           20.10.8
 API version:       1.41
 Go version:        go1.16.6
 Git commit:        3967b7d
 Built:             Fri Jul 30 19:53:57 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d8
  Built:            Fri Jul 30 19:52:06 2021
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.4.9
  GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
 runc:
  Version:          1.0.1
  GitCommit:        v1.0.1-0-g4144b63
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.6.1)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 8
  Running: 8
  Paused: 0
  Stopped: 0
 Images: 16
 Server Version: 20.10.8
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e25210fe30a0a703442421b0f60afac609f950a3
 runc version: v1.0.1-0-g4144b63
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.13.5-051305-generic
 Operating System: Ubuntu 21.04
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 47.01GiB
 Name: cheekon
 ID: Q5DS:AESB:UPDQ:ZCAM:6TTS:UUG2:DDCV:ZXK2:6LAM:LLSI:4K56:G7PP
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
 Live Restore Enabled: false

And for aarch64:

docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.6.1)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 25
 Server Version: 20.10.8
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e25210fe30a0a703442421b0f60afac609f950a3
 runc version: v1.0.1-0-g4144b63
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.10.52-v8+
 Operating System: Debian GNU/Linux 11 (bullseye)
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 7.63GiB
 Name: rpi4b
 ID: EPIN:334T:BCVZ:RY5G:II3S:YFNA:O3D4:MJHI:AAVX:MTMA:3UMF:DZGA
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true

 Live Restore Enabled: false

WARNING: No memory limit support
WARNING: No swap limit support

docker version:

Client: Docker Engine - Community
 Version:           20.10.8
 API version:       1.41
 Go version:        go1.16.6
 Git commit:        3967b7d
 Built:             Fri Jul 30 19:55:21 2021
 OS/Arch:           linux/arm64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d8
  Built:            Fri Jul 30 19:53:34 2021
  OS/Arch:          linux/arm64
  Experimental:     true
 containerd:
  Version:          1.4.9
  GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
 runc:
  Version:          1.0.1
  GitCommit:        v1.0.1-0-g4144b63
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Additional environment details (AWS, VirtualBox, physical, etc.):

I can't seem to get the platform feature working to run arm/v7 on the aarch64 host either.

Loosely following the instructions here: https://www.docker.com/blog/multi-platform-docker-builds/

Using the same Dockerfile:

FROM debian:buster
CMD uname -m

Initial setup:

curl -Ls https://github.com/docker/buildx/releases/download/v0.6.1/buildx-v0.6.1.linux-arm64 -o ~/.docker/cli-plugins/docker-buildx
chmod +x  ~/.docker/cli-plugins/docker-buildx
docker pull tonistiigi/binfmt
docker run --privileged --rm tonistiigi/binfmt --uninstall qemu-*
docker run --privileged --rm tonistiigi/binfmt --install all

Then:

docker buildx build --platform linux/arm/v7 -t arm-build . --load
[+] Building 0.7s (5/5) FINISHED
 => [internal] load build definition from Dockerfile                                                                 0.0s
 => => transferring dockerfile: 133B                                                                                 0.0s
 => [internal] load .dockerignore                                                                                    0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [internal] load metadata for docker.io/library/debian:buster                                                     0.2s
 => CACHED [1/1] FROM docker.io/library/debian:buster@sha256:cc58a29c333ee594f7624d968123429b26916face46169304f0758  0.0s
 => exporting to image                                                                                               0.2s
 => => exporting layers                                                                                              0.0s
 => => writing image sha256:ea14390799f31110d3d42dcbd41a97bf1ff57ae3392595335be9b8d1db358fc0                         0.2s
 => => naming to docker.io/library/arm-build                                                                         0.0s

Now let's try to run this arm/v7 container:

docker run --rm arm-build
WARNING: The requested image's platform (linux/arm/v7) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
aarch64

That doesn't seem right. Let's force platform?

docker run --platform linux/arm/v7 --rm arm-build
aarch64

That doesn't look right either.

docker inspect arm-build | grep Architecture
        "Architecture": "arm",

That seems right, so maybe this is a docker problem and not a build problem?

docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS  PLATFORMS
default * docker
  default default         running linux/arm64, linux/arm/v7, linux/arm/v6, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386

That seems right, no?

Here's more info about the system:

Running on RPi4B with RPI kernel and current docker:
uname -a:
Linux rpi4b 5.10.52-v8+ #1441 SMP PREEMPT Tue Aug 3 18:14:03 BST 2021 aarch64 GNU/Linux

Originally reported at docker/buildx#725

[satmandu]

This could be a dupe of #611

[satmandu]

Essentially setarch $archname should be an entrypoint that --platform $archname invokes when $archname is a member of the list setarch --list.

Once should not have to have setarch installed in the container.

[thaJeztah]

Will need to have a closer look, but some of this could be related to the normalizing code in containerd (used for platform matching) https://github.com/containerd/containerd/blob/f2c3122e9c6470c052318497899b290a5afc74a5/platforms/platforms.go#L167

/cc @tonistiigi @tianon

[tonistiigi]

This does not look like a parsing issue but what emulator/fallback reports. uname -a returns host arch from system, so either what emulator generates or if you are running 32bit x86 in 64bit system it runs natively. setarch is not used.

[tianon]

A couple PRs very relevant to the setarch request are opencontainers/runtime-spec#1012 (spec support for Personality) and opencontainers/runc#3126 (runc support for the same, which is currently not merged yet).

The way I usually deal with this (for builds of 32bit arches in https://github.com/docker-library/official-images) is to run a separate Docker-in-Docker that uses linux32 (setarch) to force the CPU to be reported as 32bit. It's not perfect, but it works (if the daemon / containerd and thus runc have the 32bit personality already, then the containers inherit it).

I'd also share the following quote that occurred recently in a Debian-related context, but I think is very relevant here too:

Is 'uname -m' a safe way for a source package to determine what architecture it's being built for?

no, the kernel architecture has nothing to do with the system architecture (chroots, qemu, 64bit kernels for 32bit userspace), which has nothing to do with the architecture it's being built for (crosscompiling)

The primary tools I use for detecting the userspace architecture (as opposed to the kernel architecture) are dpkg --print-architecture, apk --print-arch, dpkg-architecture (depending on what it's for -- this is really helpful when you want to pass an appropriate value to ./configure and is even available in Alpine), or if all else fails, something like objdump --file-headers on a binary in the image (which should then have an architecture value).

[tonistiigi]

If you want to determine the current arch in Dockerfile use TARGETARCH TARGETPLATFORM build-args.