Difficulties in using Linux-sandbox on Ubuntu 24

[sluongng]

Description of the bug:

There seems to be a set of issues with "linux-sandbox" strategy when running on Ubuntu 24.10. There were reports on Slack that this was also seen on 24.04 so I am not sure when was this the problem exactly.

The sandbox is blocked by AppArmor

It seems like the latest AppArmor profile for unprivileged_userns will block the mount operations by default.

sudo dmesg | grep -m1 'DENIED.*linux-sandbox'
[   91.164199] audit: type=1400 audit(1729776490.609:257): apparmor="DENIED" operation="mount" class="mount" info="failed mntpnt match" error=-13 profile="unprivileged_userns" name="/" pid=4261 comm="linux-sandbox" flags="rw, rprivate"

The workaround is to manually create a profile that matches the linux-sandbox binary and allow everything.

> cat /etc/apparmor.d/linux-sandbox
# This profile allows everything and only exists to give the
# application a name instead of having the label "unconfined"

abi <abi/4.0>,
include <tunables/global>

profile linux-sandbox /home/<my-user>/.cache/bazel/_bazel_<my-user>/install/*/linux-sandbox xattrs=(security.apparmor="trusted") flags=(unconfined) {
  userns,

  # Site-specific additions and overrides. See local/README for details.
  include if exists <local/linux-sandbox>
}

after that, set the xattr with sudo setfattr -n 'security.apparmor' -v 'trusted' <path-to-sandbox> for the rule to match the binary.

Finally, reload the AppArmor with sudo apparmor_parser -r /etc/apparmor.d/linux-sandbox.

Note that if the path in profile name is not strict enough (i.e. too many glob, not using xattr, etc... ) then it's possible for AppArmor to fallback to unprivileged_userns profile instead. To check run .../linux-sandbox '/bin/true' && sudo dmesg | tail -20.

I think this is an insane level of gymnastic to set up the host for linux-sandbox to work. Especially when the binary came from Bazel releases, which is managed through Bazelisk. I wonder if we should add something to Bazelisk that would install the AppArmor exclusion automatically?

--experimental_use_hermetic_linux_sandbox does not work.

When you try to run bazel with this flag turn on, you will quickly run into this error message

src/main/tools/linux-sandbox-pid1.cc:548: "execvp(/bin/bash, 0x25f93f90)": No such file or directory

I think this is caused by /bin/bash on ubuntu being a dynamically linked binary

> ldd /bin/bash
        linux-vdso.so.1 (0x00007b7dd4c50000)
        libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007b7dd4a8f000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007b7dd4800000)
        /lib64/ld-linux-x86-64.so.2 (0x00007b7dd4c52000)

so without /lib/x86_64-linux-gnu/ being available, basic commands won't work.
We can "ask" the user to add these mounts manually... but I think a sane default should be provided when linux-sandbox is used?

--sandbox_debug does not work

When troubleshooting all these, I was trying to leverage the debug flag to reproduce the issues.
However, I quickly ran into these errors

src/main/tools/linux-sandbox-pid1.cc:98: "open": File exists
...
src/main/tools/linux-sandbox-pid1.cc:110: "link dev/null": File exists

This is because the sandbox does not cleanup paths under tmp and dev after each debug runs. So subsequent debug command (copy pasted) would fail to recrate these paths. The workaround here is to clean these paths up manually before each debug copy-paste runs.

rm -f /<some-path>/e213eb56b5ca50a8b9a30b5b16a4e504/sandbox/linux-sandbox/11/tmp/empty_file
rm -f /<some-path>/e213eb56b5ca50a8b9a30b5b16a4e504/sandbox/linux-sandbox/11/dev/*

I think the linux-sandbox code should tolerate these special files from pre-existing to enable smoother debugging use cases.

Which category does this issue belong to?

Local Execution

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

No response

Which operating system are you running Bazel on?

Ubuntu 24.10

What is the output of bazel info release?

release 7.4.0

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

I think there should be a saner default provided for linux-sandbox option. Or we should mark the option as "warning" or "deprecated" to better signal the support status of the technology.

[fmeum]

Re --experimental_use_hermetic_linux_sandbox:
I think that this is working as intended: /bin/bash is a non-hermetic system dependency, so every user of the hermetic sandbox needs to decide between allowing it (and its transitive deps) and using a hermetic replacement.

Re "The sandbox is blocked by AppArmor":
The linux-sandbox binary, if given an AppArmor exception in whatever way, fully bypasses the change to AppArmor defaults that disallows unprivileged user namespaces: any unprivileged binary binary on the system can run linux-sandbox -R binary and then operate in a user namespace where it is root. For this reason, we should probably just recommend disabling the new restriction entirely, everything else would just give users a false sense of security.