Skip to content

If namespacing fails, give clearer errors as to why - #2290

Merged
palfrey merged 1 commit into
TraceMachina:mainfrom
palfrey:clearer-namespace-errors
Apr 30, 2026
Merged

palfrey merged 1 commit into
TraceMachina:mainfrom
palfrey:clearer-namespace-errors

Conversation

@palfrey

@palfrey palfrey commented Apr 29, 2026 •

Copy link
Copy Markdown
Member

Description

Primarily built to aid debug of #2173 (comment) but investigation noted that it just doesn't work inside Docker!

We previously had 3 different locations all saying "Namespaces not supported" with no clarity as to why. This PR clarifies the reason in two cases (not running Linux), and in the 3rd case it adds logging for each of the possible failure paths.

Type of change

Please delete options that aren't relevant.

  • New feature (non-breaking change which adds functionality)

How Has This Been Tested?

bazel test //... and adding use_namespaces to the integration tests

Checklist

  • Updated documentation if needed
  • Tests added/amended
  • bazel test //... passes locally
  • PR is contained in a single commit, using git amend see some docs

This change is Reviewable

@palfrey
palfrey force-pushed the clearer-namespace-errors branch 2 times, most recently from 795cc9b to d9cea5a Compare April 29, 2026 15:54
@palfrey
palfrey force-pushed the clearer-namespace-errors branch from d9cea5a to 1eb363e Compare April 29, 2026 15:56
@palfrey
palfrey marked this pull request as ready for review April 29, 2026 16:38
chrisstaite-menlo

This comment was marked as outdated.

@palfrey palfrey left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All of the println's in the pid == 0 are now replaced with exit code magic, and the privileged container situation has been documented

@palfrey made 9 comments.
Reviewable status: 0 of 1 LGTMs obtained, and 4 of 6 files reviewed, and 7 discussions need to be resolved.


nativelink-config/src/cas_server.rs line 862 at r1 (raw file):

Previously, chrisstaite-menlo (Chris Staite) wrote…

Might be worth noting that this is only true if they aren't --priviledged

Added a comment about that


nativelink-worker/src/namespace_utils.rs line 100 at r1 (raw file):

Previously, chrisstaite-menlo (Chris Staite) wrote…

This is not async-signal-safe.

I've replaced all these with specific exit codes and a print outside of the pid == 0 section. I was trying something with https://docs.rs/signal-hook/latest/signal_hook/low_level/channel/struct.Channel.html to send full messages across the barrier, but then realised I could do fun things with exit codes and bitshifting instead


nativelink-worker/src/namespace_utils.rs line 102 at r1 (raw file):

Previously, chrisstaite-menlo (Chris Staite) wrote…

This is not async-signal-safe.

Same as the other ones


nativelink-worker/src/namespace_utils.rs line 107 at r1 (raw file):

Previously, chrisstaite-menlo (Chris Staite) wrote…

This is not async-signal-safe.

Same as the other ones


nativelink-worker/src/namespace_utils.rs line 109 at r1 (raw file):

Previously, chrisstaite-menlo (Chris Staite) wrote…

This is not async-signal-safe.

Same as the other ones


nativelink-worker/src/namespace_utils.rs line 114 at r1 (raw file):

Previously, chrisstaite-menlo (Chris Staite) wrote…

This is not async-signal-safe.

Same as the other ones


nativelink-worker/src/namespace_utils.rs line 115 at r1 (raw file):

Previously, chrisstaite-menlo (Chris Staite) wrote…

This is not async-signal-safe.

Same as the other ones


nativelink-worker/src/namespace_utils.rs line 119 at r1 (raw file):

Previously, chrisstaite-menlo (Chris Staite) wrote…

This is not async-signal-safe.

Same as the other ones

error!(errno, "Namespaces: Error during unshare");
if errno == libc::EPERM {
error!(
"If the worker is inside Docker, namespaces don't work unless it's a privileged container"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very helpful improvement IMO

@chrisstaite-menlo chrisstaite-menlo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chrisstaite-menlo reviewed 5 files and all commit messages, made 1 comment, and resolved 7 discussions.
Reviewable status: 0 of 1 LGTMs obtained, and all files reviewed, and 1 discussions need to be resolved.


web/platform/src/content/docs/docs/config/production-config.mdx line 588 at r2 (raw file):

Note this only applies for non-Dockerised workers, as workers in Docker can't make the new user namespace as they're
already in a [chroot](https://en.wikipedia.org/wiki/Chroot). If however you use privileged containers, this works as

This is not the reason why, it's because they don't have the permission to create a nested user namespace. https://man7.org/linux/man-pages/man7/user_namespaces.7.html

@palfrey palfrey left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

chroot -> notes about user namespace permissions

@palfrey made 3 comments.
Reviewable status: 0 of 1 LGTMs obtained, and 5 of 8 files reviewed, and pending CI: Redis store tester, macos-26, ubuntu-24.04, integration-tests, ubuntu-24.04 / stable, windows-2022 / stable, pre-commit-checks, Analyze (javascript-typescript), Analyze (python), Coverage, Publish image, Publish nativelink-worker-init, Publish nativelink-worker-lre-cc, Publish nativelink-worker-lre-rs, Web Platform Deployment / macos-26, Web Platform Deployment / ubuntu-24.04, Local / lre-cc / ubuntu-24.04, Local / lre-rs / macos-26, Local / lre-rs / ubuntu-24.04, Bazel Dev / macos-26, Bazel Dev / ubuntu-24.04, Cargo Dev / macos-26, Cargo Dev / ubuntu-24.04, Installation / macos-26, Installation / ubuntu-24.04, buck2, buildstream, mongo, rbe-toolchain, asan / ubuntu-24.04, CodeQL, and 1 discussions need to be resolved.


web/platform/src/content/docs/docs/config/production-config.mdx line 588 at r2 (raw file):

Previously, chrisstaite-menlo (Chris Staite) wrote…

This is not the reason why, it's because they don't have the permission to create a nested user namespace. https://man7.org/linux/man-pages/man7/user_namespaces.7.html

Fixed

@palfrey
palfrey force-pushed the clearer-namespace-errors branch from 049ded8 to 99939f7 Compare April 30, 2026 15:26
@palfrey
palfrey force-pushed the clearer-namespace-errors branch from 99939f7 to 95d9bde Compare April 30, 2026 16:15
@palfrey
palfrey merged commit 3ae7168 into TraceMachina:main Apr 30, 2026
36 of 46 checks passed
@palfrey
palfrey deleted the clearer-namespace-errors branch April 30, 2026 18:05
rejuvenile pushed a commit to rejuvenile/nativelink that referenced this pull request Aug 19, 2026

This branch was previously deployed

1 inactive deployment
production — 95d9bded Deployed Apr 30, 2026 by palfrey via Web Platform Deployment / ubuntu-24.04 #2405
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants