runc 1.3.3 fails in unprivileged Proxmox LXC with "permission denied" on sysctl

[joshuafuller]

runc 1.3.3 fails in unprivileged Proxmox LXC with "permission denied" on sysctl

Description

After upgrading to runc 1.3.3-0ubuntu1~22.04.2, all Docker containers fail to start in unprivileged Proxmox LXC containers with nesting enabled. The error occurs when runc attempts to access /proc/sys/net/ipv4/ip_unprivileged_port_start through a detached procfs mount, which AppArmor interprets as accessing /sys/net/ipv4/ip_unprivileged_port_start and denies.

This appears to be related to CVE-2025-52881 and the security fix that introduced detached procfs mounts to prevent mount race attacks. The issue affects production systems running Docker in Proxmox LXC containers.

Error:

OCI runtime create failed: runc create failed: unable to start container process:
error during container init: open sysctl net.ipv4.ip_unprivileged_port_start file:
reopen fd 8: permission denied: unknown

Steps to reproduce the issue

1. Create an unprivileged Proxmox LXC container with Ubuntu 22.04:

   pct create 114 local:vztmpl/ubuntu-22.04-standard_22.04-1_amd64.tar.zst \
     --features nesting=1,keyctl=1 \
     --unprivileged 1 \
     --memory 2048 \
     --cores 4

2. Install Docker inside the LXC container:

   apt-get update
   apt-get install -y docker.io docker-compose

3. Upgrade runc to 1.3.3:

   apt-get update
   apt-get install runc=1.3.3-0ubuntu1~22.04.2

4. Try to start any Docker container:

   docker run --rm hello-world

Describe the results you received and expected

Expected result:
Container starts successfully.

Actual result:

docker: Error response from daemon: failed to create task for container:
failed to create shim task: OCI runtime create failed: runc create failed:
unable to start container process: error during container init:
open sysctl net.ipv4.ip_unprivileged_port_start file:
reopen fd 8: permission denied: unknown.

All containers fail with exit code 126 (permission denied).

Docker logs show:

failed to start container: failed to create task for container:
failed to create shim task: OCI runtime create failed: runc create failed:
unable to start container process: error during container init:
open sysctl net.ipv4.ip_unprivileged_port_start file:
reopen fd 8: permission denied: unknown

What version of runc are you using?

Failing version:

runc version 1.3.3-0ubuntu1~22.04.2
spec: 1.2.1
go: go1.23.1
libseccomp: 2.5.3

Working version (after downgrade):

runc version 1.1.0-0ubuntu1
spec: 1.0.2-dev
go: go1.17.3
libseccomp: 2.5.3

Host OS information

PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian

Container Configuration:

• Proxmox VE 8.2.2
• Unprivileged LXC container
• Features: nesting=1, keyctl=1
• OS Type: ubuntu

Host kernel information

Linux TeslaMate 6.8.4-3-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.4-3 (2024-05-02T11:55Z) x86_64 x86_64 x86_64 GNU/Linux

Additional Information

Timeline of Issue

• 2025-11-06 06:07 UTC: Ubuntu's unattended-upgrade automatically upgraded runc from 1.1.12 to 1.3.3
• 2025-11-06 ~07:56 UTC: After LXC container restart, all Docker containers failed to start

Root Cause Analysis

The security fix in runc 1.3.3 uses detached procfs mounts to prevent mount race attacks. When runc tries to write to /proc/sys/net/ipv4/ip_unprivileged_port_start, AppArmor's path-based security model misinterprets the detached mount path as /sys/net/... which is blocked by Proxmox LXC's AppArmor profile.

Attempted Fixes (Unsuccessful)

1. Modified AppArmor profile /etc/apparmor.d/abstractions/lxc/container-base:

   • Changed deny /sys/[^fdc]* to deny /sys/[^fdcn]* (to allow /sys/net)
   • Added explicit rule: /sys/net/** rw,
   • Added: /proc/sys/net/ipv4/ip_unprivileged_port_start w,
   • Reloaded profiles with apparmor_parser -r /etc/apparmor.d/lxc-containers
   • Result: Still failed with same error

2. Set sysctl on host level:

   sysctl -w net.ipv4.ip_unprivileged_port_start=0

   • Result: Did not resolve the issue

Working Workaround

Downgrade to runc 1.1.0:

apt-get install -y --allow-downgrades runc=1.1.0-0ubuntu1
apt-mark hold runc

Note: This workaround loses the security fixes from runc 1.3.3 (including fixes for 3 container escape vulnerabilities).

Related Issues

This issue has been reported in multiple projects:

• nested docker 28.5.2 unable start containers; net.ipv4.ip_unprivileged_port_start permission denied lxc/incus#2623 - Incus team implemented an AppArmor fix
• [Docker, Inc.'s package] net.ipv4.ip_unprivileged_port_start permission denied only on 1.7.28-2~debian.13~trixie containerd/containerd#12484 - containerd discussion
• CVE-2025-52881: fd reopening causes issues with AppArmor profiles (open sysctl net.ipv4.ip_unprivileged_port_start file: reopen fd 8: permission denied) #4968 - CVE-2025-52881 tracking
• https://forum.proxmox.com/threads/docker-inside-lxc-net-ipv4-ip_unprivileged_port_start-error.175437/ - Proxmox forum

The Incus fix (PR #2624) modified AppArmor to allow /sys/net access, but this same fix doesn't appear to work in Proxmox LXC environments.

Impact

This affects all users running:

• Docker in unprivileged Proxmox LXC containers with nesting
• Any container runtime using runc 1.3.3 in similar environments
• Production systems that receive automatic security updates

Questions

1. Is the AppArmor fix approach the correct solution for Proxmox LXC?
2. Should runc handle the AppArmor denial more gracefully?
3. Is there a way to make runc skip this sysctl modification when it's already appropriately set at the host level?
4. Should there be a runtime flag to disable this specific sysctl modification for nested container environments?

System Details

• Docker: 27.5.1
• containerd: 1.7.24
• docker-init: 0.19.0
• Proxmox VE: 8.2.2
• LXC container: unprivileged with nesting=1, keyctl=1

[joshuafuller]

Additional Data: AppArmor Fixes Don't Work in Proxmox LXC

I can confirm this issue affects Proxmox LXC environments, and I've tested all the recommended AppArmor fixes - none of them work.

Environment

• Host: Proxmox VE 8.2.2
• Container: Unprivileged LXC (Ubuntu 22.04.5)
• Features: nesting=1, keyctl=1
• Kernel: 6.8.4-3-pve
• Docker: 27.5.1
• containerd: 1.7.24
• Failing runc: 1.3.3-0ubuntu1~22.04.2

Timeline

• 2025-11-06 06:07 UTC: Ubuntu's unattended-upgrade automatically upgraded runc 1.1.12 → 1.3.3
• 2025-11-06 ~08:00 UTC: All Docker containers failed to start after LXC restart

Error

OCI runtime create failed: runc create failed: unable to start container process:
error during container init: open sysctl net.ipv4.ip_unprivileged_port_start file:
reopen fd 8: permission denied: unknown

AppArmor Fixes Attempted (All Failed)

Following the recommendations from this thread and lxc/incus#2624, I modified /etc/apparmor.d/abstractions/lxc/container-base:

1. Modified character class (as suggested by @cyphar)

# Changed from:
deny /sys/[^fdc]*{,/**} wklx,
# To:
deny /sys/[^fdcn]*{,/**} wklx,

Result: Still fails with same error

2. Added explicit /sys/net allow rule

/sys/net/** rw,
deny /sys/[^fdcn]*{,/**} wklx,

Result: Still fails with same error

3. Added explicit sysctl allow rule

/proc/sys/net/ipv4/ip_unprivileged_port_start w,
deny /proc/sys/net?*{,/**} wklx,

Result: Still fails with same error

4. Commented out ALL general /sys deny rules (lines 195-230)

# Disabled all these rules:
# deny /sys/[^fdcn]*{,/**} wklx,
# deny /sys/c[^l]*{,/**} wklx,
# ... (all 35+ lines of /sys deny rules)

Result: Still fails with same error!

After each change, I:

1. Reloaded AppArmor: apparmor_parser -r /etc/apparmor.d/lxc-containers
2. Restarted the LXC container: pct stop 114 && pct start 114
3. Attempted to start Docker containers

Verification

• LXC container shows unconfined AppArmor profile: cat /proc/1/attr/current → unconfined
• Can manually write to sysctl: echo 0 > /proc/sys/net/ipv4/ip_unprivileged_port_start → Success
• Docker containers use docker-default AppArmor profile

Despite all these fixes, runc 1.3.3 still fails with the same permission denied error.

Working Workaround

Only solution: Downgrade to runc 1.1.0

apt-get install -y --allow-downgrades runc=1.1.0-0ubuntu1
apt-mark hold runc

Analysis

The fact that:

1. The Incus fix (PR incusd/apparmor/lxc: Don't bother with sys/proc protections when nest… lxc/incus#2624) works for Incus/LXD
2. The same fix doesn't work for Proxmox LXC
3. Even removing all /sys deny rules doesn't help

...suggests that Proxmox LXC's AppArmor implementation differs significantly from Incus/LXD, and the detached procfs mount interaction is handled differently.

Questions for Maintainers

1. Are there known differences in how Proxmox LXC handles AppArmor vs Incus/LXD?
2. Is there additional AppArmor configuration specific to Proxmox that needs modification?
3. Should runc detect when it's in a nested/restricted environment and skip this sysctl modification?
4. Could this be a Docker-specific AppArmor profile issue (docker-default) rather than the LXC profile?

Impact

This affects all Proxmox users running Docker in unprivileged LXC containers with nesting - a very common setup for production environments. The automatic upgrade via unattended-upgrade makes this particularly disruptive.

Currently, downgrading is the only working solution for Proxmox LXC users, which means losing the security fixes from runc 1.3.3.

[spaghetti-coder]

Currently, downgrading is the only working solution for Proxmox LXC users, which means losing the security fixes from runc 1.3.3.

Confirm that. Plus lxc.apparmor.profile: unconfined in my case, which also requires for ubuntu containers apt remove -y apparmor && systemctl restart docker.service, otherwise

Error response from daemon: Could not check if docker-default AppArmor profile was loaded: open /sys/kernel/security/apparmor/profiles: permission denied

Also apparmor comes as a recommended dependency to docker.io and who knows what else, so it becomes really annoying for automated provisioning. I.e. I can't just uninstall apparmor on LXC-container setup, install docker and run containers, because docker installation brings apparmor back. Unless --no-install-recommends, but it's another headache to split an ansible role for ubuntu and everything else. And again need to cross fingers hoping no other installed tool depends on apparmor.

[cyphar]

1. This is a duplicate of CVE-2025-52881: fd reopening causes issues with AppArmor profiles (open sysctl net.ipv4.ip_unprivileged_port_start file: reopen fd 8: permission denied) #4968.
2. Docker is requesting the sysctl, runc is just applying the requested configuration. We can't just ignore it.
3. The issue has definitely been narrowed down to being caused by this particular design flaw of AppArmor. I don't see any information in your (very long) comments that makes me think the situation is any different.
4. LXC sets up AppArmor nesting, even if you see unconfined inside the container you will find that from the host the process is actually confined by AppArmor.

However, there is some Proxmox-specific stuff going on. As far as I can see, proxmox configures all containers with lxc.apparmor.profile = generated. Looking at https://github.com/lxc/lxc/blob/v6.0.5/src/lxc/lsm/apparmor.c#L1069, this causes LXC to generate a new profile based on hardcoded rules they have embeded in their binary.

This is why reloading AppArmor profiles has no effect -- you can try adding a deny /foo wklx rule to /etc/apparmor.d/abstractions/container-base and reloading the profile (with systemctl restart apparmor or apparmor_parser -r /etc/apparmor.d/abstractions/lxc-containers) and you'll find that it actually doesn't apply to the containers. So the issue here is not that the mitigation I listed doesn't work, it's that you aren't actually applying it.

Setting lxc.apparmor.profile = lxc-container-default-with-nesting or lxc.apparmor.profile = unconfined in /etc/pve/lxc/114.conf should work but then you get the errors when trying to access /sys/kernel/security/apparmor/profiles. That being said, aa-exec -p lxc-container-default-with-nesting cat /sys/kernel/security/apparmor/profiles gives an error, so maybe that wouldn't work regardless. The fact that lxc.apparmor.profile = unconfined doesn't work is quite strange though.

At this point I would use retsnoop to try to figure out where the permission errors are coming from, but it seems that Proxmox does not have enough debug stuff enabled in their kernels for this to work. At this point I think you need to ask the Proxmox folks to take a look at this.

Only solution: Downgrade to runc 1.1.0

That's really quite overkill. First of all, runc 1.1.x is completely unsupported and 1.1.0 is a 4 year old release. runc 1.2.7 and 1.3.2 do not contain these particular security fixes but are at least patched against other issues.

Does disabling AppArmor enitrely (aa-disable) not work? Even that is less drastic than downgrading to runc 1.1.0. 👀

[cyphar]

Okay, the permission error for /sys/kernel/security/apparmor/profiles is coming from aa_policy_view_capable -- this an LSM restriction that is not directly related to the profile you are using but is instead a generic policy of AppArmor. My reading is that the problem is that the AppArmor label level is not equal to the userns level (my god this logic is wacky...):

/**
 * aa_policy_view_capable - check if viewing policy in at @ns is allowed
 * @subj_cred: cred of subject
 * @label: label that is trying to view policy in ns
 * @ns: namespace being viewed by @label (may be NULL if @label's ns)
 *
 * Returns: true if viewing policy is allowed
 *
 * If @ns is NULL then the namespace being viewed is assumed to be the
 * tasks current namespace.
 */
bool aa_policy_view_capable(const struct cred *subj_cred,
			     struct aa_label *label, struct aa_ns *ns)
{
	struct user_namespace *user_ns = subj_cred->user_ns;
	struct aa_ns *view_ns = labels_view(label);
	bool root_in_user_ns = uid_eq(current_euid(), make_kuid(user_ns, 0)) ||
			       in_egroup_p(make_kgid(user_ns, 0));
	bool response = false;
	if (!ns)
		ns = view_ns;

	if (root_in_user_ns && aa_ns_visible(view_ns, ns, true) &&
	    (user_ns == &init_user_ns ||
	     (unprivileged_userns_apparmor_policy != 0 &&
	      user_ns->level == view_ns->level)))
		response = true;

	return response;
}

AFAICS this is why unconfined doesn't work -- the policy here is applied regardless of whether you are unconfined in the top-level (AppArmor policy stack) namespace. This seems like another AppArmor bug to me, but I guess I can theoretically understand the argument against making unconfined user namespaces be able to see host profiles...

The generated profiles don't hit this because they are correctly nested. This unforuntately needs to be fixed within LXC (using a very similar patch to lxc/incus#2624).

Hotfix

As a hotfix, you can trick Docker into thinking that AppArmor is disabled on the system, and then everything will work fine. Based on my tests, this command (inside the LXC container) fixes the issue:

% mount --bind /dev/null /sys/module/apparmor/parameters/enabled
% systemctl restart docker

This is in combination with setting lxc.apparmor.profile=unconfined in /etc/pve/lxc/CTR.conf. In theory lxc.apparmor.profile=lxc-container-default-with-nesting should work but it has issues with overlayfs in my testing.

[FernandoRD]

For ubuntu 24.04.3: apt-get install -y --allow-downgrades containerd.io=1.7.28-1ubuntu.24.04noble

[Resmo94]

Will be solved via a fix from docker?

[cyphar]

@Resmo94 Please read the top comment in #4968. The fix needs to come from LXC or Proxmox, and Proxmox developers just posted a patch.

[muttleydosomething]

It took me days to find and understand what was happening to my LXC containers, I thought I'd been hacked and in a way I suppose we all have been. If a hacker could penetrate the Linux development team and then write some code that would look innocent enough to make it through all checks, and then take down literally millions of production servers world-wide... This is absolutely unbelievable, it's the sort of thing you would expect from Microsoft.