cgroups v2 support - possible with systemd/freeipa-container ? #429
Comments
|
FreeIPA container works just fine with cgroups v2 if your system actually provides cgroups v2. I have no problems with Fedora 34+ and podman, with both root and rootless runs. |
|
Thanks for the swift response. Going to do some further testing and feed back. |
|
Running on a cgroups v2 enabled environment (ubuntu 21.10): and running FreeIPA as per docs: Adding or removing the -v /sys/fs/cgroup:/sys/fs/cgroup:ro option makes no difference, nor does changing the freeipa-container version back to freeipa/freeipa-server:fedora-32 (I tried a few versions). I thought this might be related to systemd 248 issues as referenced here: https://www.mail-archive.com/systemd-devel@lists.freedesktop.org/msg46076.html |
|
Try partial tests as described in the debugging section of the README: https://github.com/freeipa/freeipa-container#debugging. It looks like you don't even get to run anything from FreeIPA itself, you have basic problem of running a systemd-based container. |
|
I tried the test as advised, however it didn't provide any insight to the systemd error. From https://github.com/freeipa/freeipa-container#debugging : |
|
Could you try those |
|
@adelton |
|
What happens if you remove "-v /sys/fs/cgroup:/sys/fs/cgroup:ro"? |
|
@fcami I tried removing Should I take it then that debian11-based hosts can not currently use cgroups v2 with freeipa-container? |
|
It seems like latest dockers can start on cgroups v2 hosts but cannot run systemd in the container. So the thing to resolve is -- how do you run systemd in containers on those machines? |
|
This is very similar to failures under K3s that @kevin-leong reported in #426, also on Ubuntu 21.10. You might want to join forces to figure out how system is supposed to be run in containers on those hosts. |
|
Ok, I'll take that as meaning that freeipa-container does not support Debian-based cgroup v2 systems. If the service runs out of the box as per documentation with Fedora/CentOS/RHEL cgroup v2 systems, but doesn't run at all with Debian-based cgroup v2 systems, that is definitely worth flagging up in the documentation. While I appreciate the reference to the other open issue, which I had previously noted, it's outside current project scope to find fixes for secondary upstream components (systemd), unfortunately... |
|
Following some of what's at https://serverfault.com/questions/1053187/systemd-fails-to-run-in-a-docker-container-when-using-cgroupv2-cgroupns-priva , I was able to get further in the execution using these options: However, I'm not sure yet of the implications of running with these options! |
|
@danryu Can you try just And if it still fails, add |
|
As for that Unable to determine the amount of available RAM message, I assume there might be some more details in the ipaserver-install.log but I wonder if a |
|
@danryu 'unable to determine the amount of available RAM' is another sign of improperly configured cgroups. Can you provide the |
|
@adelton we have two places that produce this error message: one inside container detection part which handles both cgroups v1 and v2, one outside it, when everything else didn't allow us to detect the amount of memory available. If it is the former, then this definitely means host misconfiguration. If it is the latter, it means systemd tools were unable to detect that we run in container and this also means host misconfiguration. |
|
@abbra |
|
Based on traceback it failed in the container detection part. The installer is looking for both /sys/fs/cgroup/memory.current and /sys/fs/cgroup/memory.max to exist. |
|
@danryu as @rcritten said, we expect that for cgroups v2 memory controller is present and accessible. You can get more details in the kernel's documentation: https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html#memory. But outside container I don't get those various To me it looks like your container runtime is not really configuring cgroup v2 memory controller. What container runtime do you really use? |
|
I'm running vanilla Ubuntu 21.10 x64, with the following vanilla docker installation: I haven't done any configuration of the docker runtime installation. |
|
Ok, you need to ensure that your kernel runs with unified cgroups hierarchy ( |
|
This feels quite bleeding edge! My expectation was that running freeipa-container would be similar to the procedure outlined in the docs. It's not yet clear to me why the experience between CentOS and Debian11 as a host OS should have such different setup requirements. It seems that the cgroups v2 issue causes many more issues than anticipated. EDIT: FreeIPA is the current project's desired solution. If we are able to mutually find a solid configuration for Debian11/cgroups_v2, that would be optimal. |
|
I think you probably need to re-adjust your project's expectations. There is currently no support beyond this forum for freeipa-container whatsoever, regardless where you are running it. That support is mostly to let you either figure out whether a problem you see is reproducible in a non-containerized deployment of FreeIPA (and thus can be filed as a bug for FreeIPA and its components) or a bug/misconfiguration in your container runtime. RHEL IdM -- as a supported solution -- currently is not supported to run in containers in RHEL 8. RHEL 7 version of IPA container is in a limited support scope. This is mostly due to state of containerization with systemd being in active development -- cgroup v2 is one feature that was absolutely required to enable secure container separation for systemd workloads, for example. You can run FreeIPA container on cgroup v1 in a traditional root-based setup but multiple of those container instances would be technically able to step over each other if broken because containers != isolation on the host. cgroup v2 support is a kernel level feature. If your k8s deployment wants to use one, you need to be prepared to enable it everywhere where it is required (and that is not only in the kernel). Others are using freeipa-container with Docker and other container runtimes but we typically hear mostly about issues with initial runtime configuration, like yours. Once they have figured out this step, we rarely hear their experience beyond 'now it works'. |
|
Thanks @abbra for the clarification on support levels across versions. It's very helpful to have your insight into the state of play across these various components, as it is clear that there are multiple moving targets, making it difficult for a relative newcomer to figure out exact requirements. With that in mind, it does seem that the configuration is very close - I'll feed back with further results. |
As far as I know, nobody has replicated this so far on Debian11-based systems. The steps below are the configuration I applied to a Debian11/Ubuntu21.10 OS in order to get freeipa-container to work. Unfortunately, after applying all the config below, I still receive the same error from the generic startup command: Configuration I followed all preparatory steps at: https://rootlesscontaine.rs/getting-started/common/
Possible issue with dbus, described as "typically needed for using systemd and cgroup v2".... |
|
I'd really recommend you start with Ubuntu-based container image and just attempt to run systemd in that container via Only once you get that working it will make sense to add Fedora and FreeIPA on Fedora to the mix. Going with rawhide is then the last step. |
|
@adelton Following your advice I have been testing using the Dockerfile definitions at I have obtained similar results between the debian10, ubuntu-20.04, and fedora-33 images. Showing below the build and run with fedora-33. |
|
Please retry without |
Yes, without --privileged, nothing runs. |
I have confirmed this now myself, and have also got the same results on Ubuntu21.10 with podman, but not docker. I can't get the Docker runtime to run any kind of systemd/cgroups_v2 combination, on Fedora/Debian/Ubuntu etc. In each case, the test setup is:
I tested: |
|
@abbra Thanks to you and the other devs for the tips. Regarding your comments about container runtimes, and in light of the tests between docker/podman above, is there anything to be done, other than log an upstream bug in Docker runtime (not sure this is an option)? As far as I can ascertain, the Docker runtime is not playing nice between cgroups v2 and systemd, whereas other runtimes have no issues. Note: Running containerd in Kubernetes, rather than the Docker runtime, is likely to circumvent this problem - tbc |
|
I was wrestling with this same problem (Debian Bullseye platform,
What worked for me was: i.e. add and then running with e.g.: i.e. This is all with a Debian guest VM running on a Debian host, using the Debian-packaged version of |
|
Using |
danryu
changed the title
|
I assume the general conclusion is, since it works with podman and does not with docker, there's still something docker needs to do to make cgroups v2 usable in containers. There's likely not much we can do to help from FreeIPA containerization point of view, so I'm closing this issue. Feel free to add information about new developments, especially if docker starts working in some version. |
|
Seems reasonable, although for completeness it'd be nice to link to the underlying Docker issue ticket for tracking. (is it opencontainers/runc#2315? or moby/moby#42910? IDK what the currently relevant internal component is or precisely what the problem freeipa hits is….) |
|
There is nothing FreeIPA specific there, it all stems from running systemd in the container (with v2 cgroups). |
|
docker/for-mac#6288 is an upstream report of this issue |
|
For the record, on Ubuntu 22.04 which comes with cgroups v2 by default, I was able to get the container running with docker.io when I enabled user namespace remapping, with no explicit |
|
I have discovered two workarounds for this issue that effectively retain all features of unified
Not perfect, next option is better IMO.
After implementing these steps, you can run a container with Docker using Please note that the information provided above applies specifically to CentOS Stream release 9 with |
It seems that podman podman composite is the current solution. |
Lack of cgroups v2 support is now a hard blocker in a number of production use-cases, where eg Debian11 as a Kubernetes node OS is mandated and v2 is required by the network manager.
Is FreeIPA container ever going to support cgroups v2, and if so, when?
EDIT: FreeIPA container DOES support cgroups v2. The issue (as revealed in the thread) is Docker runtime failing to configure systemd correctly between host/container on cgroups v2 host systems.
podman and other container runtimes are expected to work correctly with cgroups v2 and FreeIPA.
The text was updated successfully, but these errors were encountered: