Cargo panics on Arch Linux when compiling on exFAT partition

[axoking]

Cargo version

cargo 1.86.0 (adf9b6ad1 2025-02-28)

Problem

Whenever I try to cargo run or cargo build a project on my exFAT partition on Arch Linux, it fails with the following error:

   Compiling test-crate v0.1.0 (/run/mount/Data/test-crate)
warning: error copying object file `/run/mount/Data/test-crate/target/debug/deps/test_crate-171d3f4b5eb94014.1844j56bgs10zwdjmbmp4ewcl.rcgu.o` to incremental directory as `/run/mount/Data/test-crate/target/debug/incremental/test_crate-1cwjhrbi82tge/s-h75ofrd5kj-040rlag-working/1844j56bgs10zwdjmbmp4ewcl.o`: Operation not permitted (os error 1)

warning: error copying object file `/run/mount/Data/test-crate/target/debug/deps/test_crate-171d3f4b5eb94014.1xv4e2iaijt2221ugfejfkj9p.rcgu.o` to incremental directory as `/run/mount/Data/test-crate/target/debug/incremental/test_crate-1cwjhrbi82tge/s-h75ofrd5kj-040rlag-working/1xv4e2iaijt2221ugfejfkj9p.o`: Operation not permitted (os error 1)

warning: error copying object file `/run/mount/Data/test-crate/target/debug/deps/test_crate-171d3f4b5eb94014.6p6gfwaw87bt5ts5a6awdvy49.rcgu.o` to incremental directory as `/run/mount/Data/test-crate/target/debug/incremental/test_crate-1cwjhrbi82tge/s-h75ofrd5kj-040rlag-working/6p6gfwaw87bt5ts5a6awdvy49.o`: Operation not permitted (os error 1)

warning: error copying object file `/run/mount/Data/test-crate/target/debug/deps/test_crate-171d3f4b5eb94014.6q31c3ffz1ao19rvlsb4tomhl.rcgu.o` to incremental directory as `/run/mount/Data/test-crate/target/debug/incremental/test_crate-1cwjhrbi82tge/s-h75ofrd5kj-040rlag-working/6q31c3ffz1ao19rvlsb4tomhl.o`: Operation not permitted (os error 1)

warning: error copying object file `/run/mount/Data/test-crate/target/debug/deps/test_crate-171d3f4b5eb94014.8wg0ifnylxkm8sglnzmusehrx.rcgu.o` to incremental directory as `/run/mount/Data/test-crate/target/debug/incremental/test_crate-1cwjhrbi82tge/s-h75ofrd5kj-040rlag-working/8wg0ifnylxkm8sglnzmusehrx.o`: Operation not permitted (os error 1)

warning: error copying object file `/run/mount/Data/test-crate/target/debug/deps/test_crate-171d3f4b5eb94014.9edjogq1wwbwf6yxpzh8o80lw.rcgu.o` to incremental directory as `/run/mount/Data/test-crate/target/debug/incremental/test_crate-1cwjhrbi82tge/s-h75ofrd5kj-040rlag-working/9edjogq1wwbwf6yxpzh8o80lw.o`: Operation not permitted (os error 1)

warning: `test-crate` (bin "test-crate") generated 6 warnings
error: failed to link or copy `/run/mount/Data/test-crate/target/debug/deps/test_crate-171d3f4b5eb94014` to `/run/mount/Data/test-crate/target/debug/test-crate`

Caused by:
  Operation not permitted (os error 1)

Running with root permissions
When I use sudo cargo run (and the project has more files than the default main.rs), it panics like this:

   Compiling test-crate v0.1.0 (/run/mount/Data/test-crate)
warning: hard linking files in the incremental compilation cache failed. copying files instead. consider moving the cache directory to a file system which supports hard linking in session dir `/run/mount/Data/test-crate/target/debug/incremental/test_crate-3aiikf0zz4qco/s-h75olu36wq-0bx3gll-working`

error: cached cgu 3jc4qbatsnqtmfg9qzi0sbrfc should have an object file, but doesn't


thread 'coordinator' panicked at /usr/src/debug/rust/rustc-1.86.0-src/compiler/rustc_codegen_ssa/src/back/write.rs:1680:29:
/usr/src/debug/rust/rustc-1.86.0-src/compiler/rustc_codegen_ssa/src/back/write.rs:1680:29: worker thread panicked
stack backtrace:
   0:     0x7cd9e10dd6b0 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h1a61e653684797d4
   1:     0x7cd9e112c6e3 - core::fmt::write::hef3878caa31fc074
   2:     0x7cd9e10d1db3 - <unknown>
   3:     0x7cd9e10dd502 - <unknown>
   4:     0x7cd9e10dfabf - <unknown>
   5:     0x7cd9e10df92a - std::panicking::default_hook::hf5b5ddbe229b04b3
   6:     0x7cd9dddd059d - <unknown>
   7:     0x7cd9e10e05f3 - std::panicking::rust_panic_with_hook::h85e74030df1c120f
   8:     0x7cd9e0b29ce5 - <unknown>
   9:     0x7cd9e0b231d9 - std[425a963a3e5d7a57]::sys::backtrace::__rust_end_short_backtrace::<std[425a963a3e5d7a57]::panicking::begin_panic<alloc[4540294945132585]::string::String>::{closure#0}, !>
  10:     0x7cd9e0a25cbd - std[425a963a3e5d7a57]::panicking::begin_panic::<alloc[4540294945132585]::string::String>
  11:     0x7cd9e0baf799 - <unknown>
  12:     0x7cd9e0ba866f - <unknown>
  13:     0x7cd9e0ba862e - <unknown>
  14:     0x7cd9e0baf6b3 - rustc_middle[a294d822c310247b]::util::bug::bug_fmt
  15:     0x7cd9de16fd5e - std[425a963a3e5d7a57]::sys::backtrace::__rust_begin_short_backtrace::<<rustc_codegen_llvm[ac121063ebf01a90]::LlvmCodegenBackend as rustc_codegen_ssa[2746cc10987e5ed]::traits::backend::ExtraBackendMethods>::spawn_named_thread<rustc_codegen_ssa[2746cc10987e5ed]::back::write::start_executing_work<rustc_codegen_llvm[ac121063ebf01a90]::LlvmCodegenBackend>::{closure#5}, core[f10772f7cf65ba1b]::result::Result<rustc_codegen_ssa[2746cc10987e5ed]::back::write::CompiledModules, ()>>::{closure#0}, core[f10772f7cf65ba1b]::result::Result<rustc_codegen_ssa[2746cc10987e5ed]::back::write::CompiledModules, ()>>
  16:     0x7cd9de08857c - <unknown>
  17:     0x7cd9e10e9cab - <unknown>
  18:     0x7cd9dc6a370a - <unknown>
  19:     0x7cd9dc727aac - <unknown>
  20:                0x0 - <unknown>

error: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.86.0 (05f9846f8 2025-03-31) (Arch Linux rust 1:1.86.0-1) running on x86_64-unknown-linux-gnu

note: compiler flags: --crate-type lib -C embed-bitcode=no -C debuginfo=2 -C incremental=[REDACTED]

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
end of query stack
warning: `test-crate` (lib) generated 1 warning
error: could not compile `test-crate` (lib) due to 1 previous error; 1 warning emitted

My system:

This only happens on my exFAT partition, not on the normal linux partition. It's mounted like this in the fstab:

LABEL=Data              /run/mount/Data exfat           rw,relatime,fmask=0000,dmask=0000

Every file has rwxrwxrwx permissions. Also, I'm able to build and run projects on that partition using my windows installation.

How to reproduce

I have no idea if you are able to reproduce it like this, but maybe give it a try:

• Use arch linux
• Get yourself an exFAT partition, and add it to the fstab with a 0000 mask and mountpoint somewhere under /run
• cd into the partition and create a new crate using cargo new
• create an empty lib.rs file in the src directory
• Go sudo cargo run and you should get the error

Stackoverflow post

Here's the link to a stackoverflow post where I asked about this:
https://stackoverflow.com/questions/79612554/cargo-says-operation-not-permitted-when-compiling-on-arch-linux

[moxian]

The compiler ideally shouldn't panic of course, but even if we silence the panic gracefully, you're still left with no useful artifacts.

As a workaround, you can disable incremental compilation in cargo config (docs) - either globally (by putting it in $HOME) or just for your exFAT drive (by putting it at the root of said drive). However disabling incremental compilation would increase build times when iterating the code, so that's probably not ideal.

Hardlinks being unsupported on exFAT is expected, but I'm surprised that you are getting errors while copying files. I assume if you do regular cp [from] [to] for the artifacts in the error message, then it will succeed?
This might be a low value info, but can you maybe run rustc src/main.rs (on a project with more than a single file, so as to trigger the error) under strace, and see what are the syscalls that getting you EPERM?.. and how those are different from cp maybe..

@rustbot label: +A-incr-comp +A-filesystem

[moxian]

Also see this error message:

warning: hard linking files in the incremental compilation cache failed. copying files instead. consider moving the cache directory to a file system which supports hard linking

suggesting you put target-dir onto a different drive https://doc.rust-lang.org/cargo/reference/build-cache.html#build-cache
(this solution - while workable - has its own drawbacks. Namely, you are likely to encounter spurious rebuilds when working on several projects at the same time, if both have a same dependency but with different features enabled. Unless you manually configure each of said projects to use different target dir to avoid conflicts)

[axoking]

When copying the file using the following command:

cp /run/mount/Data/test-crate/target/debug/deps/test_crate-5586b82572a0a3b2.3jc4qbatsnqtmfg9qzi0sbrfc.rcgu.o /run/mount/Data/test-crate/target/debug/incremental/test_crate-3aiikf0zz4qco/s-h75vcwn08y-132y3b5-working/3jc4qbatsnqtmfg9qzi0sbrfc.o

I get this error message because the file doesn't exist:

cp: cannot stat '/run/mount/Data/test-crate/target/debug/deps/test_crate-5586b82572a0a3b2.3jc4qbatsnqtmfg9qzi0sbrfc.rcgu.o': No such file or directory

so it seems like Cargo is trying to copy a file that does not exist.

To your suggestion about running rustc src/main.rs: it compiles the hello world file and I can also execute it. I guess this is because rustc just compiles that single file while cargo does more stuff around that. However when using cargo rustc, I get the same errors as before.

When running strace cargo run, I don't get any lines with the word "EPERM".

But I will have a look at your suggestion using a different target-dir, thank you

[saethlin]

I guess this is because rustc just compiles that single file while cargo does more stuff around that.

Cargo is enabling incremental compilation by default in debug builds by passing an extra flag to rustc. If you invoke the compiler manually, you are not passing the required flag.

Following moxian's suggestion to disable incremental compilation should also let the build go through.