Skip to content

toolchains: don't make an unpreservable file mode a build failure - #1580

Open
akafael wants to merge 1 commit into
bazel-contrib:mainfrom
akafael:fix/copy-out-preserve-attributes
Open

toolchains: don't make an unpreservable file mode a build failure#1580
akafael wants to merge 1 commit into
bazel-contrib:mainfrom
akafael:fix/copy-out-preserve-attributes

Conversation

@akafael

@akafael akafael commented Aug 13, 2026

Copy link
Copy Markdown

Fixes #1579

What

replace_symlink emits cp -a for every declared binary and library output, and symlink_to_dir
uses cp -pR. Both -a and -p set coreutils' require_preserve, which promotes a failed
attribute copy from a warning to exit 1. When the destination filesystem refuses the chmod that
--preserve=mode performs, cp writes the contents correctly and then fails:

cp: preserving permissions for 'bazel-out/.../lib/libFoo.so': Operation not permitted

The generated wrapper runs under set -euo pipefail, so the action dies with a correct artifact
already on disk.

This replaces both copies with cp -R plus a separate touch -r, which is what
copy_dir_contents_to_dir has done since #583 ("Use touch not cp -p to preserve timestamps") —
these two call sites were missed. Applied to all four toolchain command files (linux, macos,
freebsd, windows); test/expected/inner_fun_text{,_macos,_freebsd}.txt are updated to match.

Two details worth calling out for review:

  • Why not cp -R --preserve=timestamps. It fixes the reported error, but --preserve= still
    sets require_preserve, so it stays fatal in the adjacent case where the destination pre-exists
    owned by another uid (cp: preserving times for '…': Operation not permitted), and BSD cp on
    macOS/FreeBSD has no --preserve= at all. cp -R + touch -r is uniform across all four
    toolchains and matches the existing idiom.
  • No touch -r in symlink_to_dir. That branch is guarded by
    [[ -L "$source" && ! -d "$source" ]], reachable only for dangling symlinks and symlinks to
    non-regular files, since a symlink to a real file already matched -f above. cp -R recreates
    the link exactly as cp -pR did, and touch -r would fail on a dangling link.

-a also implies -d; at the replace_symlink call site the symlink has just been resolved with
readlink -f (realpath on macOS) and removed, so the source is a real file or directory, -R
covers the directory case, and the no-dereference behaviour is not relied on.

This is a strict narrowing of what is preserved, so it cannot break a build that previously
succeeded. The only behaviour lost is mode/ownership/ACL propagation into bazel-out, which Bazel
does not guarantee in the first place.

Validating the mechanism (10 seconds, no cluster needed)

That -p/-a are what make this fatal reproduces on any GNU coreutils. Copying onto a
destination whose chmod fails:

rc=1  cp -a                           :: cp: preserving permissions for '…': Operation not permitted
rc=1  cp -pR                          :: cp: preserving permissions for '…': Operation not permitted
rc=1  cp -p                           :: cp: preserving permissions for '…': Operation not permitted
rc=1  cp -R --preserve=mode           :: cp: preserving permissions for '…': Operation not permitted
rc=1  cp -R --preserve=ownership      :: cp: preserving permissions for '…': Operation not permitted
rc=0  cp -R --preserve=timestamps     :: <clean>
rc=0  cp -R                           :: <clean>

Note a non-root uid alone is not sufficient — on ext4/overlayfs, cp -a from a root-owned
source to a fresh destination as an unprivileged uid exits 0, because the failed chown is
tolerated by chown_failure_ok() and the chmod succeeds on the file cp just created. The
necessary condition is a destination filesystem that rejects chmod.

Validating end to end with a local Buildbarn

bb-deployments gives a full cluster from Docker
Compose. Its FUSE worker materialises the action directory — including bazel-out — on
bb_worker's virtual filesystem rather than a plain local one, which is the interesting case.

1. Start the cluster

git clone https://github.com/buildbarn/bb-deployments
cd bb-deployments/docker-compose
./run.sh -d          # needs sudo: sets up volumes and a FUSE mount on the host
docker compose ps    # frontend, scheduler, storage-{0,1}, worker+runner (fuse, hardlinking)

2. Have the worker report what the action directory actually is

genrule(
    name = "fsprobe",
    outs = ["fsprobe.log"],
    cmd = """{
      id
      stat -f -c 'bazel-out fstype=%T' .
      echo x > p
      chmod 0755 p && echo "chmod: ok" || echo "chmod: FAILED"
      cp -a p q; echo "cp -a  exit=$$?"
      cp -R p r; echo "cp -R  exit=$$?"
    } > $@ 2>&1""",
)

platform(
    name = "rbe_ubuntu2204",
    exec_properties = {
        "OSFamily": "linux",
        "container-image": "docker://ghcr.io/catthehacker/ubuntu:act-22.04@sha256:dd7654ffb01d5b7b54b23b9ce928a1f7f2d08c7b3d7e320b6574b55d7ccde78b",
    },
    parents = ["@platforms//host"],
)
bazel build //:fsprobe \
  --remote_executor=grpc://localhost:8980 \
  --remote_instance_name=fuse \
  --extra_execution_platforms=//:rbe_ubuntu2204
cat bazel-bin/fsprobe.log

cp -a exit=1 with cp -R exit=0 is the failure this PR fixes.
--remote_instance_name=hardlinking selects the worker that uses a plain local directory instead,
which is a useful contrast.

3. Reproduce through the rules

replace_symlink fires whenever a declared output is a symlink, which is the normal result of a
CMake project that sets SOVERSION — the versioned links are installed straight into the declared
output directory:

add_library(demo SHARED demo.c)
set_target_properties(demo PROPERTIES VERSION 0.9.1 SOVERSION 0)
install(TARGETS demo LIBRARY DESTINATION lib)
cmake(name = "demo", lib_source = ":srcs", out_shared_libs = ["libdemo.so"])

Build it with the same remote flags, against this branch and against main. On main the action
fails at the cp -a block at the end of bazel-bin/.../build_script.sh, after CMake.log shows
the install completing. The generated block can also be run standalone under set -euo pipefail
to isolate it from the rest of the build.

Testing

  • The coreutils bisection above: run and reproduced.
  • bazel test //test/... on Linux: 94/94 pass, including //test:shell_script_inner_fun_test,
    which diffs the generated script against the regenerated test/expected/ goldens.
  • The Buildbarn steps are written from the bb-deployments cluster config
    (worker-fuse-ubuntu22-04.jsonnet, runner-ubuntu22-04.jsonnet); I have not run the full cmake
    reproduction through it end to end, so treat step 3 as instructions rather than a recorded
    result.

I could not find a way to assert this failure in CI without a second uid or a special filesystem,
so the change is covered by the existing golden and shellcheck tests rather than a new regression
test. Happy to add one if you have a preferred mechanism.

`replace_symlink` emits `cp -a` for every declared binary and library
output, and `symlink_to_dir` uses `cp -pR`. Both `-a` and `-p` set
coreutils' `require_preserve`, which promotes a failed attribute copy
from a warning to exit 1. When the destination filesystem refuses the
`chmod` that `--preserve=mode` performs, `cp` writes the file contents
correctly and then fails:

    cp: preserving permissions for 'bazel-out/.../lib/libFoo.so': Operation not permitted

The generated wrapper runs under `set -euo pipefail`, so the action dies
with a correct artifact already on disk. This is reachable from any
CMake project that sets SOVERSION, since the versioned symlinks are
installed straight into the declared output directory.

Copy the data and restore the timestamp separately, which is what
`copy_dir_contents_to_dir` already does since bazel-contrib#583 ("Use touch not cp -p
to preserve timestamps"); these two call sites were missed. `cp -R`
still covers the symlink-to-directory case that `-a`'s implied `-d`
handled, and the source at the `replace_symlink` call site is already
resolved by `readlink -f`/`realpath`, so no-dereference is not relied
on.

No `touch -r` in `symlink_to_dir`: that branch is guarded by
`[[ -L "$source" && ! -d "$source" ]]` and is reachable only for
dangling symlinks and symlinks to non-regular files, where `touch -r`
would itself fail.

This is a strict narrowing of what is preserved, so it cannot break a
build that previously succeeded. The only behaviour lost is
mode/ownership/ACL propagation into `bazel-out`, which Bazel does not
guarantee.

Refs: bazel-contrib#1579
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

replace_symlink's cp -a makes an unpreservable file mode a hard build failure

1 participant