nerdctl can not run command when inside the nerdctl containers

[Huimintai]

I want to use the nerdctl runcommand into the nerdctl container but I found the nerdctl runcan not work:

1. Create the nerd container:

nerdctl run  -d --privileged --net=host --restart=always -v /etc/hosts:/app/hosts -v /run/containerd/containerd.sock:/run/containerd/containerd.sock -v /opt/tke-installer/data:/app/data -v /opt/tke-installer/conf:/app/conf -v registry-certs:/app/certs -v tke-installer-bin:/app/bin docker.io/library/test:20 tke-installer

2. Exec itto the nerd container:

# nerdctl exec -it 09850b2176a9 sh

3. Run one container:

/app # nerdctl images | grep alpine
alpine                                     latest           234cb88d3020    2 days ago           5.9 MiB

/app # nerdctl run  alpine:latest
FATA[0000] no such file or directory

But I can run the container outside the container in my host:

root@VM-8-15-ubuntu:~#  nerdctl run  alpine:latest
root@VM-8-15-ubuntu:~#

So does the nerdctl support run containers into the nerd container?

/app # nerdctl version
Client:
 Version:	v0.9.0-12-g72ac023
 Git commit:	72ac023383304d57947f75ce2077d3952aacd0e0

Server:
 containerd:
  Version:	v1.5.2
  Revision:	36cc874494a56a253cd181a1a685b44b58a2e34a

I use the master branch build.

[AkihiroSuda]

Mounting /run/containerd/containerd.sock isn't enough.
At least you need to mount the following directories, and maybe more

• /run/containerd
• /var/lib/containerd
• /var/lib/nerdctl
• /tmp (probably)

[Huimintai]

Hi @AkihiroSuda what the default bind mount mode?Does it is read write?I still can not create the container into nerdctl container,this is the error log:

# nerdctl logs 02dcf0a6e552
FATA[0000] failed to open "/var/lib/nerdctl/1935db59/containers/default/02dcf0a6e552bd7b898dac96173d1650eadd40e5858bf6e22fd412f0eafab7be/02dcf0a6e552bd7b898dac96173d1650eadd40e5858bf6e22fd412f0eafab7be-json.log", container is not created with `nerdctl run -d`?: stat /var/lib/nerdctl/1935db59/containers/default/02dcf0a6e552bd7b898dac96173d1650eadd40e5858bf6e22fd412f0eafab7be/02dcf0a6e552bd7b898dac96173d1650eadd40e5858bf6e22fd412f0eafab7be-json.log: no such file or directory

It can not create the log file when I mount the /var/lib/nerdctl into the container use -v /var/lib/nerdctl/:/var/lib/nerdctl/
For docker volume inspect I can see the volume mode rw but for nerdctl volume inspect I can not find the volume mode.

This is my command:

nerdctl run  -d --privileged --net=host --restart=always -v /etc/hosts:/app/hosts -v /run/containerd:/run/containerd -v /var/lib/containerd:/var/lib/containerd -v /var/lib/nerdctl:/var/lib/nerdctl -v /opt/cni/bin:/opt/cni/bin -v /run/containerd/containerd.sock:/run/containerd/containerd.sock -v /opt/tke-installer/data:/app/data -v /opt/tke-installer/conf:/app/conf -v registry-certs:/app/certs -v tke-installer-bin:/app/bin docker.io/library/test:20 tke-installer

create the registry-http container command:

nerdctl run -d  --name registry-http -p 80:5000 -v /opt/tke-installer/registry:/var/lib/registry  registry.tke.com/library/registry-amd64:2.7.1

Also I can create the log file into the container:

/var/lib/nerdctl/1935db59/containers/default/3b7c20e63a99b618d57d0366f40fabdbc80dee0556183f040bd6019ecf510520 # echo > 3b7c20e63a99b618d57d0366f40fabdbc80dee055
6183f040bd6019ecf510520-json.log

[Huimintai]

When I mount the /var/lib/nerdctl I can not create the containers:

root@VM-8-15-ubuntu:~# nerdctl run  -d --privileged --net=host --restart=always -v /var/lib/nerdctl:/var/lib/nerdctl -v /opt/cni/bin:/opt/cni/bin -v /run/containerd:/run/containerd -v /var/lib/containerd:/var/lib/containerd -v /etc/hosts:/app/hosts -v /run/containerd/containerd.sock:/run/containerd/containerd.sock -v /opt/tke-installer/data:/app/data -v /opt/tke-installer/conf:/app/conf -v registry-certs:/app/certs -v tke-installer-bin:/app/bin test:21 tke-installer
9cca347c94496213f1d18c45b3abf94ab7217b0b71fc35551eadcf006205fd99
root@VM-8-15-ubuntu:~#
root@VM-8-15-ubuntu:~# nerdctl ps
CONTAINER ID    IMAGE                        COMMAND                   CREATED          STATUS    PORTS    NAMES
9cca347c9449    docker.io/library/test:21    "/app/bin/tke-instal…"    2 seconds ago    Up
root@VM-8-15-ubuntu:~# nerdctl exec -it 9cca347c9449 sh
/app #
/app #
/app # nerdctl run ubuntu:latest
FATA[0000] failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #0:: fork/exec /usr/bin/nerdctl: no such file or directory: unknown
/app #
/app # nerdctl -v
nerdctl version 0.9.0-12-g72ac023
/app # nerdctl ps
CONTAINER ID    IMAGE                        COMMAND                   CREATED           STATUS    PORTS    NAMES
9cca347c9449    docker.io/library/test:21    "/app/bin/tke-instal…"    50 seconds ago    Up
/app #

[AkihiroSuda]

Bind-mounting the containerd socket and misc dirs from the host is not straightforward.
So maybe better to just run another containerd in the container if it works for you.
Example:

nerdctl/Dockerfile

Line 244 in ad836c2

FROM base AS demo

[Huimintai]

@AkihiroSuda Sorry I don't understand your Example:< Do you mean I should install the containerd into the nerdctl container instead of mount the containerd socket?
Looking forward to your reply，thanks.

[AkihiroSuda]

yes

[Huimintai]

So I need build the nerdctl image first then I can use the nerdctl run to run the nerdctl containerd into the nerd container all right?

nerdctl run -it --rm --privileged nerdctl:latest

[AkihiroSuda]

yes

[m00lecule]

@AkihiroSuda @Huimintai I am facing similar issue. Do You know an elegant example how to run another containerd daemon inside nerdctl container. The official ghcr.io/containerd/nerdctl:main container is running as root user which complicates the rootless containerd daemon installation.

[arogozhnikov]

Hi @AkihiroSuda , thanks for your remark:

At least you need to mount the following directories, and maybe more ...

I'm coming from docker-land where passing just one socket is enough, and I was struggling with that (also ChatGPT convinced me I indeed need only one socket. Is there a complete list of directories I need? Above list didn't work

[ningmingxiao]

@Huimintai you can use -v /usr/bin/nerdctl:/usr/bin/nerdctl in container because shim want to find nerdctl binary as logdriver.

nerdctl run -d --privileged --net=host -v /usr/bin/nerdctl:/usr/bin/nerdctl -v /var/lib/nerdctl:/var/lib/nerdctl -v /run/containerd:/run/containerd -v /var/lib/containerd:/var/lib/containerd --network=none -d docker.io/library/busybox:1.28 sleep 10000