Failing to build when any "features" are added? Error in devContainersSpecCLI.js

[epicwhale]

Type: Bug

When I try to build with any "features" enabled using below, the build fails. When I comment out the features, it works fine.

	"features": {
		"ghcr.io/devcontainers-contrib/features/poetry:2": {}
	}

I also tried with "ghcr.io/devcontainers/features/common-utils:2": {} - same issue

Dev container failure log:

[2024-08-19T13:26:05.210Z] Dev Containers 0.381.0 in VS Code 1.93.0-insider (d751e4324d12370e950f8cd031aa8829637ce300).
[2024-08-19T13:26:05.210Z] Start: Resolving Remote
[2024-08-19T13:26:05.263Z] Setting up container for folder or workspace: /Users/#redacted#/code/#redacted#
[2024-08-19T13:26:05.283Z] Start: Check Docker is running
[2024-08-19T13:26:05.283Z] Start: Run: podman version
[2024-08-19T13:26:05.418Z] Client:       Podman Engine
Version:      5.1.1
API Version:  5.1.1
Go Version:   go1.22.3
Git Commit:   bda6eb03dcbcf12a5b7ae004c1240e38dd056d24
Built:        Tue Jun  4 22:34:52 2024
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      5.0.1
API Version:  5.0.1
Go Version:   go1.21.8
Built:        Mon Apr  1 01:00:00 2024
OS/Arch:      linux/arm64
[2024-08-19T13:26:05.419Z] Stop (136 ms): Run: podman version
[2024-08-19T13:26:05.419Z] Stop (136 ms): Check Docker is running
[2024-08-19T13:26:05.420Z] Start: Run: podman volume ls -q
[2024-08-19T13:26:05.514Z] Stop (94 ms): Run: podman volume ls -q
[2024-08-19T13:26:05.515Z] Start: Run: podman ps -q -a --filter label=vsch.local.folder=/Users/#redacted#/code/#redacted# --filter label=vsch.quality=insider
[2024-08-19T13:26:05.613Z] Stop (98 ms): Run: podman ps -q -a --filter label=vsch.local.folder=/Users/#redacted#/code/#redacted# --filter label=vsch.quality=insider
[2024-08-19T13:26:05.614Z] Start: Run: podman ps -q -a --filter label=devcontainer.local_folder=/Users/#redacted#/code/#redacted# --filter label=devcontainer.config_file=/Users/#redacted#/code/#redacted#/.devcontainer/devcontainer.json
[2024-08-19T13:26:05.708Z] Stop (94 ms): Run: podman ps -q -a --filter label=devcontainer.local_folder=/Users/#redacted#/code/#redacted# --filter label=devcontainer.config_file=/Users/#redacted#/code/#redacted#/.devcontainer/devcontainer.json
[2024-08-19T13:26:05.708Z] Start: Run: podman ps -q -a --filter label=devcontainer.local_folder=/Users/#redacted#/code/#redacted#
[2024-08-19T13:26:05.841Z] Stop (133 ms): Run: podman ps -q -a --filter label=devcontainer.local_folder=/Users/#redacted#/code/#redacted#
[2024-08-19T13:26:05.841Z] Running Dev Containers CLI:   up --user-data-folder /Users/#redacted#/Library/Application Support/Code - Insiders/User/globalStorage/ms-vscode-remote.remote-containers/data --docker-path podman --container-session-data-folder /tmp/devcontainers-e0fd13b9-dea1-4da1-b64a-2ee361c824d81724073964833 --workspace-folder /Users/#redacted#/code/#redacted# --workspace-mount-consistency cached --id-label devcontainer.local_folder=/Users/#redacted#/code/#redacted# --id-label devcontainer.config_file=/Users/#redacted#/code/#redacted#/.devcontainer/devcontainer.json --log-level debug --log-format json --config /Users/#redacted#/code/#redacted#/.devcontainer/devcontainer.json --default-user-env-probe loginInteractiveShell --remove-existing-container --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root --include-configuration --include-merged-configuration
[2024-08-19T13:26:05.858Z] ProxyResolver#loadSystemCertificates count
[2024-08-19T13:26:05.859Z] ProxyResolver#loadSystemCertificates count filtered
[2024-08-19T13:26:05.859Z] Start: Run: /Applications/Visual Studio Code - Insiders.app/Contents/Frameworks/Code - Insiders Helper (Plugin).app/Contents/MacOS/Code - Insiders Helper (Plugin) /Users/#redacted#/.vscode-insiders/extensions/ms-vscode-remote.remote-containers-0.381.0/dist/spec-node/devContainersSpecCLI.js up --user-data-folder /Users/#redacted#/Library/Application Support/Code - Insiders/User/globalStorage/ms-vscode-remote.remote-containers/data --docker-path podman --container-session-data-folder /tmp/devcontainers-e0fd13b9-dea1-4da1-b64a-2ee361c824d81724073964833 --workspace-folder /Users/#redacted#/code/#redacted# --workspace-mount-consistency cached --id-label devcontainer.local_folder=/Users/#redacted#/code/#redacted# --id-label devcontainer.config_file=/Users/#redacted#/code/#redacted#/.devcontainer/devcontainer.json --log-level debug --log-format json --config /Users/#redacted#/code/#redacted#/.devcontainer/devcontainer.json --default-user-env-probe loginInteractiveShell --remove-existing-container --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root --include-configuration --include-merged-configuration
[2024-08-19T13:26:06.010Z] @devcontainers/cli 0.67.0. Node.js v20.15.1. darwin 23.6.0 arm64.
[2024-08-19T13:26:06.010Z] Start: Run: podman buildx version
[2024-08-19T13:26:06.108Z] Stop (98 ms): Run: podman buildx version
[2024-08-19T13:26:06.108Z] buildah 1.36.0
[2024-08-19T13:26:06.108Z] 
[2024-08-19T13:26:06.108Z] Start: Run: podman -v
[2024-08-19T13:26:06.135Z] Stop (27 ms): Run: podman -v
[2024-08-19T13:26:06.135Z] Start: Resolving Remote
[2024-08-19T13:26:06.139Z] Start: Run: podman compose version --short
[2024-08-19T13:26:06.267Z] Stop (128 ms): Run: podman compose version --short
[2024-08-19T13:26:06.267Z] Docker Compose version: 2.24.6
[2024-08-19T13:26:06.267Z] Start: Run: podman compose -f /Users/#redacted#/code/#redacted#/docker-compose.yml -f /Users/#redacted#/code/#redacted#/.devcontainer/docker-compose.dev.yml --profile * config
[2024-08-19T13:26:06.397Z] Stop (130 ms): Run: podman compose -f /Users/#redacted#/code/#redacted#/docker-compose.yml -f /Users/#redacted#/code/#redacted#/.devcontainer/docker-compose.dev.yml --profile * config
[2024-08-19T13:26:06.397Z] name: #redacted#
services:
  devcontainer:
    command:
      - sleep
      - infinity
    image: mcr.microsoft.com/devcontainers/python:1-3.12-bullseye
    networks:
      default: null
    volumes:
      - type: bind
        source: /Users/#redacted#/code/#redacted#
        target: /workspace
        bind:
          create_host_path: true
networks:
  default:
    name: #redacted#_default
[2024-08-19T13:26:06.399Z] Start: Run: podman ps -q -a --filter label=com.docker.compose.project=#redacted# --filter label=com.docker.compose.service=devcontainer
[2024-08-19T13:26:06.497Z] Stop (98 ms): Run: podman ps -q -a --filter label=com.docker.compose.project=#redacted# --filter label=com.docker.compose.service=devcontainer
[2024-08-19T13:26:06.498Z] Start: Run: podman events --format json --filter event=start
[2024-08-19T13:26:06.500Z] PersistedPath=/Users/#redacted#/Library/Application Support/Code - Insiders/User/globalStorage/ms-vscode-remote.remote-containers/data, ContainerHasLabels=false
[2024-08-19T13:26:06.500Z] Start: Run: podman compose -f /Users/#redacted#/code/#redacted#/docker-compose.yml -f /Users/#redacted#/code/#redacted#/.devcontainer/docker-compose.dev.yml --profile * config
[2024-08-19T13:26:06.640Z] Stop (140 ms): Run: podman compose -f /Users/#redacted#/code/#redacted#/docker-compose.yml -f /Users/#redacted#/code/#redacted#/.devcontainer/docker-compose.dev.yml --profile * config
[2024-08-19T13:26:06.640Z] name: #redacted#
services:
  devcontainer:
    command:
      - sleep
      - infinity
    image: mcr.microsoft.com/devcontainers/python:1-3.12-bullseye
    networks:
      default: null
    volumes:
      - type: bind
        source: /Users/#redacted#/code/#redacted#
        target: /workspace
        bind:
          create_host_path: true
networks:
  default:
    name: #redacted#_default
[2024-08-19T13:26:06.640Z] Start: Run: podman inspect --type image mcr.microsoft.com/devcontainers/python:1-3.12-bullseye
[2024-08-19T13:26:06.736Z] Stop (96 ms): Run: podman inspect --type image mcr.microsoft.com/devcontainers/python:1-3.12-bullseye
[2024-08-19T13:26:06.737Z] Resolving Feature dependencies for 'ghcr.io/devcontainers-contrib/features/poetry:2'...
[2024-08-19T13:26:06.737Z] * Processing feature: ghcr.io/devcontainers-contrib/features/poetry:2
[2024-08-19T13:26:06.738Z] Loading 17 extra certificates from /var/folders/fj/34bpbrh132n6sf_l1jwy1fbr00k_x2/T/vsch/certificates-10872e31ebcca293010e55290f60274f59d64a05cfecc3264fe92df30df7bbb1.pem.
[2024-08-19T13:26:06.863Z] Start: Run: docker-credential-osxkeychain get
[2024-08-19T13:26:06.865Z] Stop (2 ms): Run: docker-credential-osxkeychain get
[2024-08-19T13:26:06.866Z] Stop (3 ms): Run: docker-credential-osxkeychain get
[2024-08-19T13:26:07.155Z] * Processing feature: ghcr.io/devcontainers-contrib/features/pipx-package
[2024-08-19T13:26:07.314Z] * Processing feature: ghcr.io/devcontainers/features/python
[2024-08-19T13:26:07.464Z] Soft-dependency 'ghcr.io/devcontainers/features/python' is not required.  Removing from installation order...
[2024-08-19T13:26:07.464Z] Soft-dependency 'ghcr.io/devcontainers-contrib/features/pipx-package' is not required.  Removing from installation order...
[2024-08-19T13:26:07.464Z] * Fetching feature: poetry_0_oci
[2024-08-19T13:26:07.655Z] * Fetched feature: poetry_0_oci version 2.0.17
[2024-08-19T13:26:07.657Z] Docker Compose override file for building image:
version: "3.8"

services:
  devcontainer:
    image: vsc-#redacted#-cd16a0e4f42995d130d53fc54afb616b8d005387da1107d279bc5e0d764e647c
    build:
      dockerfile: /var/folders/fj/34bpbrh132n6sf_l1jwy1fbr00k_x2/T/devcontainercli/container-features/0.67.0-1724073966736/Dockerfile-with-features
      context: /Users/#redacted#/Library/Application Support/Code - Insiders/User/globalStorage/ms-vscode-remote.remote-containers/data/empty-folder
      args:
        - BUILDKIT_INLINE_CACHE=1
        - _DEV_CONTAINERS_BASE_IMAGE=dev_container_auto_added_stage_label
        - _DEV_CONTAINERS_IMAGE_USER=root
        - _DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp
      additional_contexts:
        - dev_containers_feature_content_source=/var/folders/fj/34bpbrh132n6sf_l1jwy1fbr00k_x2/T/devcontainercli/container-features/0.67.0-1724073966736

[2024-08-19T13:26:07.657Z] Start: Run: podman compose --project-name #redacted# -f /Users/#redacted#/code/#redacted#/docker-compose.yml -f /Users/#redacted#/code/#redacted#/.devcontainer/docker-compose.dev.yml -f /Users/#redacted#/Library/Application Support/Code - Insiders/User/globalStorage/ms-vscode-remote.remote-containers/data/docker-compose/docker-compose.devcontainer.build-1724073967657.yml build
[2024-08-19T13:26:07.768Z] >>>> Executing external compose provider "/usr/local/bin/docker-compose". Please refer to the documentation for details. <<<<

[2024-08-19T13:26:07.795Z] the classic builder doesn't support additional contexts, set DOCKER_BUILDKIT=1 to use BuildKit
[2024-08-19T13:26:07.796Z] Error: executing /usr/local/bin/docker-compose --project-name #redacted# -f /Users/#redacted#/code/#redacted#/docker-compose.yml -f /Users/#redacted#/code/#redacted#/.devcontainer/docker-compose.dev.yml -f /Users/#redacted#/Library/Application Support/Code - Insiders/User/globalStorage/ms-vscode-remote.remote-containers/data/docker-compose/docker-compose.devcontainer.build-1724073967657.yml build: exit status 1
[2024-08-19T13:26:07.797Z] Stop (140 ms): Run: podman compose --project-name #redacted# -f /Users/#redacted#/code/#redacted#/docker-compose.yml -f /Users/#redacted#/code/#redacted#/.devcontainer/docker-compose.dev.yml -f /Users/#redacted#/Library/Application Support/Code - Insiders/User/globalStorage/ms-vscode-remote.remote-containers/data/docker-compose/docker-compose.devcontainer.build-1724073967657.yml build
[2024-08-19T13:26:07.798Z] Error: Command failed: podman compose --project-name #redacted# -f /Users/#redacted#/code/#redacted#/docker-compose.yml -f /Users/#redacted#/code/#redacted#/.devcontainer/docker-compose.dev.yml -f /Users/#redacted#/Library/Application Support/Code - Insiders/User/globalStorage/ms-vscode-remote.remote-containers/data/docker-compose/docker-compose.devcontainer.build-1724073967657.yml build
[2024-08-19T13:26:07.798Z]     at Km (/Users/#redacted#/.vscode-insiders/extensions/ms-vscode-remote.remote-containers-0.381.0/dist/spec-node/devContainersSpecCLI.js:430:525)
[2024-08-19T13:26:07.798Z]     at async QtA (/Users/#redacted#/.vscode-insiders/extensions/ms-vscode-remote.remote-containers-0.381.0/dist/spec-node/devContainersSpecCLI.js:430:2476)
[2024-08-19T13:26:07.798Z]     at async utA (/Users/#redacted#/.vscode-insiders/extensions/ms-vscode-remote.remote-containers-0.381.0/dist/spec-node/devContainersSpecCLI.js:410:3506)
[2024-08-19T13:26:07.798Z]     at async KtA (/Users/#redacted#/.vscode-insiders/extensions/ms-vscode-remote.remote-containers-0.381.0/dist/spec-node/devContainersSpecCLI.js:482:3944)
[2024-08-19T13:26:07.798Z]     at async eB (/Users/#redacted#/.vscode-insiders/extensions/ms-vscode-remote.remote-containers-0.381.0/dist/spec-node/devContainersSpecCLI.js:482:4886)
[2024-08-19T13:26:07.798Z]     at async hrA (/Users/#redacted#/.vscode-insiders/extensions/ms-vscode-remote.remote-containers-0.381.0/dist/spec-node/devContainersSpecCLI.js:663:200)
[2024-08-19T13:26:07.798Z]     at async lrA (/Users/#redacted#/.vscode-insiders/extensions/ms-vscode-remote.remote-containers-0.381.0/dist/spec-node/devContainersSpecCLI.js:662:13452)
[2024-08-19T13:26:07.800Z] Stop (1941 ms): Run: /Applications/Visual Studio Code - Insiders.app/Contents/Frameworks/Code - Insiders Helper (Plugin).app/Contents/MacOS/Code - Insiders Helper (Plugin) /Users/#redacted#/.vscode-insiders/extensions/ms-vscode-remote.remote-containers-0.381.0/dist/spec-node/devContainersSpecCLI.js up --user-data-folder /Users/#redacted#/Library/Application Support/Code - Insiders/User/globalStorage/ms-vscode-remote.remote-containers/data --docker-path podman --container-session-data-folder /tmp/devcontainers-e0fd13b9-dea1-4da1-b64a-2ee361c824d81724073964833 --workspace-folder /Users/#redacted#/code/#redacted# --workspace-mount-consistency cached --id-label devcontainer.local_folder=/Users/#redacted#/code/#redacted# --id-label devcontainer.config_file=/Users/#redacted#/code/#redacted#/.devcontainer/devcontainer.json --log-level debug --log-format json --config /Users/#redacted#/code/#redacted#/.devcontainer/devcontainer.json --default-user-env-probe loginInteractiveShell --remove-existing-container --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root --include-configuration --include-merged-configuration
[2024-08-19T13:26:07.800Z] Exit code 1
[2024-08-19T13:26:07.803Z] Command failed: /Applications/Visual Studio Code - Insiders.app/Contents/Frameworks/Code - Insiders Helper (Plugin).app/Contents/MacOS/Code - Insiders Helper (Plugin) /Users/#redacted#/.vscode-insiders/extensions/ms-vscode-remote.remote-containers-0.381.0/dist/spec-node/devContainersSpecCLI.js up --user-data-folder /Users/#redacted#/Library/Application Support/Code - Insiders/User/globalStorage/ms-vscode-remote.remote-containers/data --docker-path podman --container-session-data-folder /tmp/devcontainers-e0fd13b9-dea1-4da1-b64a-2ee361c824d81724073964833 --workspace-folder /Users/#redacted#/code/#redacted# --workspace-mount-consistency cached --id-label devcontainer.local_folder=/Users/#redacted#/code/#redacted# --id-label devcontainer.config_file=/Users/#redacted#/code/#redacted#/.devcontainer/devcontainer.json --log-level debug --log-format json --config /Users/#redacted#/code/#redacted#/.devcontainer/devcontainer.json --default-user-env-probe loginInteractiveShell --remove-existing-container --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root --include-configuration --include-merged-configuration
[2024-08-19T13:26:07.803Z] Exit code 1

Extension version: 0.381.0
VS Code version: Code - Insiders 1.93.0-insider (d751e4324d12370e950f8cd031aa8829637ce300, 2024-08-16T05:03:54.591Z)
OS version: Darwin arm64 23.6.0
Modes:

System Info

Item	Value
CPUs	Apple M1 Max (10 x 2400)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_graphite: disabled_off video_decode: enabled video_encode: enabled webgl: enabled webgl2: enabled webgpu: enabled webnn: disabled_off
Load (avg)	4, 6, 6
Memory (System)	64.00GB (1.33GB free)
Process Argv	--crash-reporter-id d287bdaf-dfb3-4405-8916-265c6f6853de
Screen Reader	no
VM	0%

A/B Experiments

vsliv368cf:30146710
vspor879:30202332
vspor708:30202333
vspor363:30204092
vswsl492cf:30256198
vscod805:30301674
vsaa593:30376534
py29gd2263:31024238
vscaac:30438845
c4g48928:30535728
962ge761:30841072
pythongtdpath:30726887
welcomedialog:30812478
pythonnoceb:30776497
asynctok:30898717
dsvsc014:30777825
dsvsc015:30821418
pythonregdiag2:30926734
pythonmypyd1:30859725
h48ei257:31000450
pythontbext0:30879054
accentitlementst:30870582
dsvsc016:30879898
dsvsc017:30880771
dsvsc018:30880772
cppperfnew:30980852
pythonait:30973460
724cj586:31013169
a69g1124:31018687
dvdeprecation:31040973
dwnewjupytercf:31046870
2f103344:31071589
impr_priority:31057980
nativerepl1:31104042
refactort:31084545
pythonrstrctxt:31093868
flighttreat:31119334
wkspc-onlycs-t:31111718
nativeloc1:31118317
wkspc-ranged-c:31118571
cf971741:31111988
fcdif617:31111928

[chrmarti]

It wrongly takes buildah 1.36.0 as the BuildKit version and assumes it can use additional build contexts (which are supported since BuildKit 0.8.0).

It seems the latest Podman and buildah should support additional build contexts, maybe podman compose is not using buildah? Have you tried setting DOCKER_BUILDKIT=1?

[CristopherH95]

I may not be the original reporter of this issue, but I am running Podman on Windows and I have been seeing this same problem. I tried setting the DOCKER_BUILDKIT environment variable to 1, but this seemed to have no effect. In my case at least, I think Podman should be using buildah because (as far as I'm aware) it is the default, and I have never attempted to change it.

[kriansa]

Using podman + docker-compose is incompatible with Buildkit, unfortunately. Podman forces DOCKER_BUILDKIT=0 when calling docker-compose.

See: https://github.com/containers/podman/blob/04ad9b67489a493898822e9015bca75d6b8280b7/cmd/podman/compose.go#L159-L164

[chrmarti]

It wrongly takes buildah 1.36.0 as the BuildKit version and assumes it can use additional build contexts (which are supported since BuildKit 0.8.0).

It seems the latest Podman and buildah should support additional build contexts, maybe podman compose is not using buildah? Have you tried setting DOCKER_BUILDKIT=1?

[CristopherH95]

I may not be the original reporter of this issue, but I am running Podman on Windows and I have been seeing this same problem. I tried setting the DOCKER_BUILDKIT environment variable to 1, but this seemed to have no effect. In my case at least, I think Podman should be using buildah because (as far as I'm aware) it is the default, and I have never attempted to change it.

[kriansa]

Using podman + docker-compose is incompatible with Buildkit, unfortunately. Podman forces DOCKER_BUILDKIT=0 when calling docker-compose.

See: https://github.com/containers/podman/blob/04ad9b67489a493898822e9015bca75d6b8280b7/cmd/podman/compose.go#L159-L164