Can't run DinD if I build my own image

[cruizba]

I'm encountering an unusual issue, and I'm unsure whether it's related to Sysbox itself or a system dependency. This problem began following an apt upgrade on my system. Below are the Dockerfile and docker-compose.yaml files I'm using:

Dockerfile

FROM ghcr.io/nestybox/ubuntu-jammy-systemd:latest

RUN apt update && apt install -y \
    sudo curl nano vim net-tools iputils-ping \
    iproute2 software-properties-common openssh-server

RUN curl -fsSL https://get.docker.com | sh

docker-compose.yaml

services:

  node:
    build: .
    runtime: sysbox-runc
    hostname: node
    container_name: node
    networks:
      network-test:
        ipv4_address: 10.10.0.5

networks:
  network-test:
    name: network-test
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 10.10.0.0/16

If I execute it...:

docker compose build
docker compose up -d
docker exec -it node docker run hello-world

... I see the following error...:

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "proc" to rootfs at "/proc": mount src=proc, dst=/proc, dstFd=/proc/thread-self/fd/8, flags=0xe: no such file or directory: unknown.

I also tried to build this Dockerfile and the same error occurs.

Interestingly, if I simply use this docker-compose.yaml everything works as expected:

services:
  node:
    image: ghcr.io/nestybox/ubuntu-jammy-systemd-docker:latest    <---- With this image it works.
    runtime: sysbox-runc
    hostname: node
    container_name: node
    networks:
      network-test:
        ipv4_address: 10.10.0.5

networks:
  network-test:
    name: network-test
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 10.10.0.0/16

This is my current system information:

Docker version

Client: Docker Engine - Community
 Version:           27.3.1
 API version:       1.47
 Go version:        go1.22.7
 Git commit:        ce12230
 Built:             Fri Sep 20 11:41:00 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.3.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.7
  Git commit:       41ca978
  Built:            Fri Sep 20 11:41:00 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.24
  GitCommit:        88bf19b2105c8b17560993bee28a01ddc2f97182
 runc:
  Version:          1.2.2
  GitCommit:        v1.2.2-0-g7cb3632
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Sysbox Version

dic 01 13:30:25 carlos-pc sh[1573]: sysbox-runc
dic 01 13:30:25 carlos-pc sh[1573]:         edition:         Community Edition (CE)
dic 01 13:30:25 carlos-pc sh[1573]:         version:         0.6.5
dic 01 13:30:25 carlos-pc sh[1573]:         commit:         1b440ff266841f3d2d296e664122a9e29ceb9fd8
dic 01 13:30:25 carlos-pc sh[1573]:         built at:         Sat Nov  9 06:09:34 UTC 2024
dic 01 13:30:25 carlos-pc sh[1573]:         built by:         Rodny Molina
dic 01 13:30:25 carlos-pc sh[1573]:         oci-specs:         1.1.0+dev
dic 01 13:30:25 carlos-pc sh[1721]: sysbox-mgr
dic 01 13:30:25 carlos-pc sh[1721]:         edition:         Community Edition (CE)
dic 01 13:30:25 carlos-pc sh[1721]:         version:         0.6.5
dic 01 13:30:25 carlos-pc sh[1721]:         commit:         1159d228eac8402efa63bd2cb18cdf9e404ea130
dic 01 13:30:25 carlos-pc sh[1721]:         built at:         Sat Nov  9 06:10:05 UTC 2024
dic 01 13:30:25 carlos-pc sh[1721]:         built by:         Rodny Molina
dic 01 13:30:25 carlos-pc sh[1730]: sysbox-fs
dic 01 13:30:25 carlos-pc sh[1730]:         edition:         Community Edition (CE)
dic 01 13:30:25 carlos-pc sh[1730]:         version:         0.6.5
dic 01 13:30:25 carlos-pc sh[1730]:         commit:         aeba775e52cc6385fa4807c594fc7ee164ad624c
dic 01 13:30:25 carlos-pc sh[1730]:         built at:         Sat Nov  9 06:10:01 UTC 2024
dic 01 13:30:25 carlos-pc sh[1730]:         built by:         Rodny Molina

Ubuntu version

Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.5 LTS
Release:	22.04
Codename:	jammy

Kernel version

6.8.0-49-generic

[dimikot]

Same error (sysbox-ce_0.6.4-0.linux).

Downgrading containerd.io to 1.7.23-1 (even inside of the docker container, not on the host) helped me.

See details here: containerd/containerd#11078 (comment)

[pguinet]

Confirm downgrading containerd.io to 1.7.23-1 inside the container works for me too.

[cruizba]

@dimikot Is it a bug or a feature? I see containerd team closed the issue.

[href]

I have the same problem - mine additionally occurs with Buildkit that I use with DIND. I had to pin Buildkit to 0.17.2 (instead of the latest 0.18.0), and override /usr/bin/runc with v1.1.15.

Note sure how I could pass any kind of parameter to work around this issue.

[pguinet]

I created an issue by containerd : containerd/containerd#11083

[DekusDenial]

I am facing the same issue without a way to resolve it. I am running a self-hosted github actions runner with sysbox, and now running the docker/build-push-action@v5 actions to build image with buildx will fail with the above same error for /proc

These are the version of docker and related in the runner image:

docker: 27.1.2
containerd: v1.7.20
runc: 1.1.13
buildx: v0.16.2

The current workaround would be to pin the version of moby buildkit image to moby/buildkit:v0.17.3 which seemingly does not have the problematic runc like above-mentioned.
@href @pguinet do you guys happen to have other suggestion for me by any chance?

[DekusDenial]

@ctalledo @rodnymolina I don't think containerd will rollback anything because the change was intentional.

[EddieX64]

Hello, we are experiencing the same issue with sysbox v0.6.4, containerd 1.7.22 and runc 1.1.7 on kubernetes nodes. It seems the buildkit version 0.18.x which was pushed yesterday has updated runc and containerd versions and pushed as stable version. I have tested creating a docker buildkit builder from the image moby/buildkit:v0.17.3 inside a sysbox pod which works fine.

docker buildx create --bootstrap --driver docker-container --name builder-test --driver-opt image=moby/buildkit:buildx-stable-1 -> NOT OK

docker buildx create --bootstrap --driver docker-container --name builder-test --driver-opt image=moby/buildkit:v0.17.3 -> OK

[href]

I am facing the same issue without a way to resolve it. I am running a self-hosted github actions runner with sysbox, and now running the docker/build-push-action@v5 actions to build image with buildx will fail with the above same error for /proc

The current workaround would be to pin the version of moby buildkit image to moby/buildkit:v0.17.3 which seemingly does not have the problematic runc like above-mentioned. @href @pguinet do you guys happen to have other suggestion for me by any chance?

@DekusDenial Without knowing all the layers of your setup it's impossible to say what would be a good approach in your setup. We use a custom setup with self-managed GitLab, and are in control of all the layers. So in our case I can just add a hack to our own images. In your case I would imagine you'd either have to somehow override docker/build-push-action@v5 to not use the latest releases, or to remove sysbox from your stack for the time being.

[ctalledo]

Hi folks, thanks for reporting, we will take a look ASAP. Likely some change in containerd or runc that is surfacing some previously hidden bug in Sysbox.