Binary releases of fd crash on Asahi Linux because of jemalloc using 8kb page sizes

[Techcable]

Binary releases crash because they don't support 16kb-aligned pages.

I get the following error with the latest binary releases (both from github releases, and the Arch Linux ARM repos):

<jemalloc>: Unsupported system page size
<jemalloc>: Unsupported system page size
memory allocation of 5 bytes failed
[1]    28699 IOT instruction (core dumped)  ./fd-v8.4.0-aarch64-unknown-linux-gnu/fd

Building from source fixes the issue (apparently jemalloc has internal logic to detect page size?)

M1 Macs use 16KB pages by default at a hardware level, so this is a common problem with Ashai Linux.

There are two simple fixes that occur to me:

1. disabling jemalloc for AArch64 Linux
2. Compiling jemalloc to use 64K pages (which will work anywhere). This is what the ALARM Rust package does.

The second one will probably have to be filled in the upstream jemallactor project (in fact that could be a more appropriate to file this issue....)

[tmccombs]

Regarding 1, Asahi/m1 isn't the only way to run linux on aarch64, so disabling jemalloc entirely for the build seems a little extreme. I don't know enough about this to know if there are downsides to 2.

Another option is to build a separate artifact for linux aarch64 without jemalloc and recommend that if you are using asahi.

[tavianator]

I think we can configure jemalloc to support 64K pages. I don't have an easy way to test it though

[Techcable]

I think we can configure jemalloc to support 64K pages. I don't have an easy way to test it though

As I understand it, a jemalloc compiled to use 64K pages supports 8K, 16K, and 64K-aligned processors at runtime (all of which are possibilities on AArch64) .

Moving the default page size to 64k (or 16k) is probably the best approach. It would keep jemalloc working in the default-compiled binaries.

Increasing the compile-time default is the approach used to fix this problem in most projects:

• rr debugger (switched to 64k)
• emacs (switched to 64k)
• Rust (packaged by Arch Linux ARM) (moved the ceiling to 16k)
  • It's not clear what upstream Rust did here
• WebKit (moved the ceiling to 64k)

Some projects (chromium, libunwind) switched to runtime detection. jemalloc upstream doesn't currently support that, so we can .

Changing the compile-time default to 64k can be done by overriding the JEMALLOC_SYS_WITH_LG_PAGE=16 environment variable of the jemalloc-sys crate (the variable is in terms of logarithms, and log2(16 * 1024) = 16)).

EDIT: I'm not sure how we would set this variable using build.rs, presumably the jemallacator crate is built before the fd crate, so build.rs couldn't really affect environment variables there.

Maybe we should fix the default upstream?

[tavianator]

EDIT: I'm not sure how we would set this variable using build.rs, presumably the jemallacator crate is built before the fd crate, so build.rs couldn't really affect environment variables there.

We could just set JEMALLOC_SYS_WITH_LG_PAGE=16 in the CI environment, and jemallocator should pick it up. I agree it would be good to report this upstream as well.

[montchr]

I can confirm that setting JEMALLOC_SYS_WITH_LG_PAGE=16 during a local build fixes the issue for me on Asahi Linux.

It would be great to have the change implemented here so Asahi Linux and other affected aarch64-linux users can make use of pre-built binaries. Is something preventing progress on this issue?

We could just set JEMALLOC_SYS_WITH_LG_PAGE=16 in the CI environment

If that's all it would take to fix this, I would be willing to open a pull request.

[tmccombs]

I would be willing to open a pull request.

@montchr that would be great.

[andradei]

Would this change be as small as setting a flag in the build step to change the page size?

[mathomp4]

I was about to make a new issue here, but I think I'm seeing the same thing on a system I'm on? I'm not on Asahi Linux, but Rocky Linux:

@$ cat /etc/os-release
NAME="Rocky Linux"
VERSION="9.3 (Blue Onyx)"
...

This is on an NVIDIA Grace-Grace node which (I think) uses 64K pages per this documentation. (I have no idea how I'd check this locally...but I'm guessing the integrator used the recommended settings for the most part.)

And when I grab the aarch64 binary and try it out:

$ fd cat
<jemalloc>: Unsupported system page size
<jemalloc>: Unsupported system page size
memory allocation of 5 bytes failed
Aborted (core dumped)

I'd gladly do more checking or testing if asked...but page sizes and all are a bit beyond my ken. :)