Wrong /dev/null permission making ubuntu jammy deployment impossible

[lanquarden]

When switching the sandbox project to deploy ubuntu jammy, running apt update with the cexec action fails due to not having permission to write to /dev/null

Expected Behaviour

Being able to run apt update when deploying the ubuntu jammy image, permissions on /dev/null need to be 666 for apt update to work.

Current Behaviour

apt update in cexec action fails when deploying the ubuntu jammy image because it can't write to /dev/null, permissions on /dev/null are 660 and apt update doesn't work

Possible Solution

First I updated cexec container to mount /dev as rw so I could update the permissions from the template. Then I switched to a more general approach where I updated hook-docker to set the correct permissions:

hook-docker/main.go
────────────────────────────────────────────────────────────────────────────────────────────────────────────

──────────────────┐
31: func main() { │
──────────────────┘
 31 ⋮ 31 │    fmt.Println("Starting Tink-Docker")
 32 ⋮ 32 │    go rebootWatch()
 33 ⋮ 33 │
    ⋮ 34 │    fmt.Println("Make /dev/null writeable for all users!")
    ⋮ 35 │    cmd := exec.Command("chmod", "666", "/dev/null")
    ⋮ 36 │    cmd.Stdout = os.Stdout
    ⋮ 37 │    cmd.Stderr = os.Stderr
    ⋮ 38 │    err := cmd.Run()
    ⋮ 39 │    if err != nil {
    ⋮ 40 │        panic(err)
    ⋮ 41 │    }
    ⋮ 42 │
 34 ⋮ 43 │    // Parse the cmdline in order to find the urls for the repository and path to the cert
 35 ⋮ 44 │    content, err := ioutil.ReadFile("/proc/cmdline")
 36 ⋮ 45 │    if err != nil {

──────────────────┐
74: func main() { │
──────────────────┘
 65 ⋮ 74 │    }
 66 ⋮ 75 │
 67 ⋮ 76 │    // Build the command, and execute
 68 ⋮    │    cmd := exec.Command("/usr/local/bin/docker-init", "/usr/local/bin/dockerd")
    ⋮ 77 │    cmd = exec.Command("/usr/local/bin/docker-init", "/usr/local/bin/dockerd")
 69 ⋮ 78 │    cmd.Stdout = os.Stdout
 70 ⋮ 79 │    cmd.Stderr = os.Stderr
 71 ⋮ 80 │    err = cmd.Run()

While I got it working I don't know if there are better ways to solve this problem.

Steps to Reproduce (for bugs)

1. Try deploying ubuntu jammy image with the sandbox

Context

Your Environment

• Operating System and version (e.g. Linux, Windows, MacOS): Linux

• How are you running Tinkerbell? Using Vagrant & VirtualBox, Vagrant & Libvirt, on Packet using Terraform, or give details: Sandbox & docker-compose deploying on bare-metal

• Link to your project or a code example to reproduce issue:

[jacobweinstock]

Hey @lanquarden, thanks for reporting this. I definitely agree that /dev/null should have 666 permissions. Weird thing is that I am able to reproduce this on one machine, but on another machine I cannot. i.e. installing the machine1 with Ubuntu Jammy works successfully.
In the successful install, /dev/null has 660 permissions. So, it makes me think that there's possibly an environmental issue external to vagrant that is causing issues. I'm definitely not certain though. I will continue to investigate and update here.

[lanquarden]

Maybe the issue resides in apt update, these commands are invoked with root user so it shouldn't complain about the 660 permission, unless apt is running some parts as a different user... Maybe this behavior is linked to the environment somehow. I didn't explore that route as I had found a workaround.

[Cajga]

@jacobweinstock we run into the same with EKS Anywhere (we opened an AWS case as well as we have EKS-A Subscription).

In fact, @lanquarden is right and by default apt is using the _apt user for calling out to apt-key (and to many other "sandboxed" tasks like downloading a package etc.) that would write to /dev/null.

The main issue here is that with the current hook kernel, when you mount /dev with devtmpfs /dev/null gets created with 0660 (and this happens inside the cexec container as well thanks to this):

While using Ubuntu 22.04 for example it gets 0666:

root@kls107:~# mkdir /mnt/dev
root@kls107:~# mount -r  -t devtmpfs none /mnt/dev
root@kls107:~# ll /mnt/dev/null
crw-rw-rw- 1 root root 1, 3 Feb  6 09:29 /mnt/dev/null
root@kls107:~# mount|grep /mnt/dev
none on /mnt/dev type devtmpfs (ro,relatime,size=65716676k,nr_inodes=16429169,mode=755,inode64)
root@kls107:~# 

We would love to get a fix (or any applicable workaround that does not require custom build cexec container/hook os) into EKS Anywhere.

[Cajga]

I've sent a PR to cexec that implements a "harmless workaround" and allows the call of apt update in the following way:

CMD_LINE: echo 'nameserver IPOFYOURNAMESERVER' > /etc/resolv.conf && export NEEDRESTART_SUSPEND=true && apt -y update && apt install -y nfs-common open-iscsi....

[jacobweinstock]

Hey @lanquarden and @Cajga , this PR should resolve this. #200 . Once its landed, or before if you want to build from my branch, would you mind testing to validate its working for you both?

[Cajga]

@jacobweinstock , thank you for the quick fix.

I can confirm that using the hook image fixed the /dev/* permissions inside hook. I can also confirm that the /dev/null has 0666 permissions inside the chrooted cexec action ( this surprised me as it is mounting devtmpfs (that had permission issues) instead of bind mounting /dev from hook).