Support k3s on LibreElec #4859
Comments
|
you can set --data-dir to change /var/lib/rancher/k3s to something else, but we don't currently have any way to relocate all of the files in /etc/rancher for non-rootless use cases. Your best bet may be to remount /etc read-write and symlink /etc/rancher to somewhere writable. |
|
Thanks for the feedback. I unfortunately can't remount / (that contains /etc) as read-write, because it's a squashfs filesystem. So I'm not able to create /etc/rancher symlink. Maybe a workaround could be to modify the systemd service, so that it runs a script with |
|
The k3s-agent service starts, creates a file /etc/rancher/node/password, but then fails on another error message |
|
That has been required by upstream Kubernetes since 1.20 if I remember correctly. Is libreelec shipping a particularly old kernel, or perhaps just doesn't have all the cgroups enabled? |
|
It uses a recent kernel : 5.10.76 (more detail in the output of |
|
Yeah, they need CONFIG_CGROUP_PIDS and probably a bunch of other ones as well. You might run |
|
Great! Many thanks for your help. Here is the output : Only CONFIG_NETFILTER_XT_MATCH_IPVS is displayed in red. |
|
The check-config script hasn't been updated in a bit. It is now required. |
|
All right. So which config options should I ask for?
|
|
I'm not 100% sure but I suspect these should do it:
|
|
I've made some progress with your help. I managed to recompile LibreELEC with these kernel options (except CONFIG_NETFILTER_XT_MATCH_IPVS, that did not seem to exist), and it solved the "PIDS cgroup support not found" issue. Anyway, after doing so (in their libreelec-10.0 branch), I managed to run it in Virtualbox, install the Docker addon, and install k3s with the following commands : I also had to modify the systemd file /storage/.kodi/addons/service.system.docker/system.d/service.system.docker.service , to make it use cgroupfs driver instead of systemd : (followed by This way, k3s-agent manages to register the node on the control-plane. Great! ... but it's not over : the service restarts in loop with the following warnings and errors : I might do a similar trick to make /usr writable (like for /etc) if necessary, at least to go further. |
|
Why are you running it with docker instead of using containerd? You might also try removing the host iptables packages, unless you have something else that needs them. K3s ships with its own fallback iptables binary that has all the required features enabled, but it is only used if the host does not already have iptables installed. That assumes the root cause is not a missing kernel module. |
Because I'm sure docker runs properly on it, as there is an official addon. AFAIK, there is no package manager to install/uninstall things : all necessary binaries seem to be compiled from scratch at build time. So it should be complicated to install containerd
I can try to disable it, by masking the systemd service (see https://github.com/LibreELEC/LibreELEC.tv/blob/master/packages/network/iptables/config/README). I just hope it's not necessary for something else in LibreELEC : they probably didn't compile and add it for no reason, but (if I'm lucky) it might be there "only" for docker. |
|
After mounting /usr in a writable place, and removing all iptables or ip6tables binaries from /usr/sbin, it seems to use the iptables bundled with k3s. It might be some missing kernel options, based on garywill/linux-router#18 and/or kubernetes-sigs/kind#1461 |
|
I added the kernel options CONFIG_NETFILTER_XT_MATCH_COMMENT and CONFIG_NETFILTER_XT_MATCH_STATISTIC But the Docker service fails to start if it does not find iptables in the path : I'll try to use the LibreELEC iptables binary, by adding some other kernel options among CONFIG_NETFILTER_XT_*MARK |
|
After many attempts, I finally made it work. I had to add many kernel options :
Here is my current procedure to run it :
|
|
I'll create a ticket/PR on LibreELEC side to ask to add the kernel options. I'll also try to make my command-lines run on each startup, to (hopefully) make it starts correctly after a reboot. Of course, this is still very dirty. |
|
Adding the following lines to /storage/.kodi/addons/service.system.docker/bin/docker-config is enough to allow k3s to start after a reboot (because it's called as NB : I can't use /etc/fstab, as this file is in the read-only filesystem at startup time. This is still quick-and-dirty :
It's enough for me to test in real conditions. |
|
K3s has containerd built in; you don't need to do anything extra to use it. Also, we're probably not going to package cri-dockerd when dockershim support is dropped in 1.24, so the --docker flag will go away. See if you can get it working without Docker; I imagine it'll be much easier. |
|
Thanks @brandond . I switched to containerd without issue. |
|
OK I manage to have a stable k3s agent (using containerd). I'm not very proud of these dirty hacks, but I'm quite happy to have achieved to make it work. I now have a different issue, though, that is probably not related to k3s : I'm using https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner to provision PVs on an external NFS server, and this is not always working. The pods seem to be able to read on NFS shares with no problem, but they fail when trying to lock a file (before writing on it). The dmesg of LibreElec repeatedly says :
I suppose it's an issue of the NFS client of LibreELEC : I'll have to investigate |
|
I found a workaround for my NFS issues. and re-created the PVCs. Now the NFS is stable, and my LibreELEC device is working great as a k3s agent. |
|
@mossroy I'm thinking about following your path as a guide, but wondering why you remounted all of /etc instead of just creating a /etc/rancher symlink on start or bind mounting just /etc/rancher? are there other files in /etc that rancher has to touch? |
|
There's no other directory that needs to be modified in /etc, except /etc/rancher. BTW I've written a blog post (in French) that might be easier to follow, as it goes straight to the solution I'm currently using (with containerd instead of docker): https://blog.mossroy.fr/2022/04/25/mediacenter-diy-saison-2-avec-libreelec-et-k3s/ |
|
It's not on the roadmap to officially support this, but it seems there's a workaround available, so I'm going to close this out. PR's are always welcome, but we don't have plans to "support" LIbreElec specifically at this time |
Is your feature request related to a problem? Please describe.
I'm trying to run a k3s agent on LibreElec (amd64), to join an existing k3s cluster.
Docker can easily be installed (there's a dedicated Kodi add-on provided by LibreElec), and works : I can run docker containers on the SSH command-line.
But this linux distribution uses unusual mounts. In particular :
So /etc is read-only, and /var is not persistent after a reboot. But /storage could be used to store both the binary and the config.
With this config, k3s does not run out-of-the-box.
NB : Systemd is available and used by this distro
Describe the solution you'd like
Ideally, I could just run the installer (with specific options if necessary) and it would run (either as server or agent).
Installation without specific options currently fails because /usr/local does not exist :
If I set
INSTALL_K3S_SYSTEMD_DIR=/storage/k3s-systemd INSTALL_K3S_BIN_DIR=/storage/k3swhen installing, the installation works :But the service does not start properly :
Describe alternatives you've considered
I've tried to run the k3s binary manually. But it fails with the same error message :
Setting
--private-registryand/or--data-dircommand-line parameters does not seem to helpAdditional context
I'd be happy to test anything
Output of
docker infoon LibreElec :Backporting
The text was updated successfully, but these errors were encountered: