BPF syscall mediation is too limited for snapd

[bboozzoo]

Required information

• Distribution: Ubuntu
• Distribution version: 24.04
• The output of "snap list --all lxd core20 core22 core24 snapd":
• The output of "lxc info" or if that fails:
  • Kernel version: 6.8.0-22-generic
  • LXC version:
  • LXD version: 5.21.1 LTS
  • Storage backend in use: (IDK used lxd init --auto)

Issue description

With snapd 2.62, when trying to run a snap using core24 base:

DEBUG: no devices tagged with snap_test-snapd-sh-core24_sh, but device cgroup is required, proceeding with setup
DEBUG: /sys/fs/bpf is not a bpffs mount
cannot mount bpf filesystem under /sys/fs/bpf: Operation not permitted

Enabling bpf mediation and bpf mount inside the container with the following settings:

  security.syscalls.intercept.bpf: "true"
  security.syscalls.intercept.bpf.devices: "true"
  security.syscalls.intercept.mount: "true"
  security.syscalls.intercept.mount.allowed: bpf

the setup can progress little further but fails when attempting to create a map:

DEBUG: device map not present yet
DEBUG: create bpf map of type 0x1, key size 9, value size 1, entries 500
cannot create bpf map: Operation not permitted

As BPF_MAP_CREATE command is rejected. Digging further, the BPF mediation code

lxd/lxd/seccomp/seccomp.go

Line 267 in 3fef7cb

static int handle_bpf_syscall(pid_t pid_target, int notify_fd, int mem_fd,

does not support many of the BPF commands we attempt.

The problem came with when microovn snap attempted the switch to core24 base. AFAIU the CI attaches a cloud-init configuration to the container which used package_upgrade: true` thus triggering an update of snapd to 2.62. However, it’s clear that the issue has existed for some time now, but apparently it was a rare occurrence to have devices tagged for a given snap inside an unprivileged

Steps to reproduce

1. lxc launch --vm ubuntu-daily:24.04 u1
2. in the vm:
   a. lxc launch ubuntu:22.04 n1
   b. refresh snapd (make sure it's 2.62+)
   c. pack and install this snap https://github.com/snapcore/snapd/tree/master/tests/lib/snaps/test-snapd-sh-core24
   i. clone snapd tree
   ii. snap pack /tests/lib/snaps/test-snapd-sh-core24
   iii. snap install --dangerous
   d. attempt SNAPD_DEBUG=1 test-snapd-sh-core24.sh -c true

for reference of all BPF commands used by snap-confine see https://github.com/snapcore/snapd/blob/83735930aefc3f4e63c528aa09798fc0b9f6a9a0/cmd/libsnap-confine-private/bpf-support.c

[tomponline]

Also see https://warthogs.atlassian.net/browse/SNAPDENG-20045

[tomponline]

c. pack and install this snap https://github.com/snapcore/snapd/tree/master/tests/lib/snaps/test-snapd-sh-core24

@bboozzoo please could you elaborate on this step, what commands are needed for this?

[tomponline]

@bboozzoo to your knowledge would we need to add any more functionality for bpf beyond BPF_MAP_CREATE ?

[bboozzoo]

I've updated the description with additional steps for packing the snap and a reference to snap confine bpf setup code.

[tomponline]

I've updated the description with additional steps for packing the snap and a reference to snap confine bpf setup code.

Thanks!

[mihalicyn]

@bboozzoo Dear Maciej,

thanks a lot for so detailed report and analysis.

The reason for this quite limited interception support in LXD is that eBPF subsystem in the Linux kernel wasn't containerized at all.
Just a year ago bpffs/maps/programs were host-wide objects without any user namespace (or any other namespace) relation.

From what I see now, situation is changing [1], [2]. It means that we can reconsider our interception support and try to come up with some permission model design, then modify our interception implementation to allow some operations to be executed from inside an unprivileged container (with interception explicitly enabled, of course).

Speaking about BPF_MAP_* operations, I can't see any problem to allow them right now, cause generally these operations are secure. (btw, if you set sysctl kernel.unprivileged_bpf_disabled=0 these operations will be allowed system-wide. But it's not fully safe, because together with BPF_MAP_* operation this will allow unprivileged BPF_PROG_LOAD without any filtering on the eBPF program type).

The big problem is that snapd relies on bpffs and we need to be really careful with this filesystem and ensure that it is well-containerized and nothing from the host leaks to the container and that containers can not affect each other through it. Yes, you can allow it to be mounted using interception right now, but I would be very-very careful with this and would not advice to do this in production.

[1] https://lore.kernel.org/bpf/20240124022127.2379740-3-andrii@kernel.org/
[2] https://lore.kernel.org/bpf/20240124022127.2379740-5-andrii@kernel.org/

Kind regards,
Alex

[tomponline]

Thanks @mihalicyn very insightful stuff.