Float-free libcore (for embedded systems and kernel drivers, among other things)

[emk]

(I was talking to @huonw about embedded Rust the other day, and he suggested I write this up as an RFC issue. I hope this is in the correct place!)

I'm having a ton of fun hacking on kernels in Rust. Rust is a wonderful fit for the problem domain, and the combination of libcore and custom JSON --target specs makes the whole process very ergonomic. But there's one issue that keeps coming up on #rust-osdev: libcore requires floating point, but many otherwise reasonable environments place restrictions on floating point use.

Existing discussions of this issue can be found here:

• Compiling libcore without SSE leads to LLVM ERROR: SSE register return with SSE disabled. If you disable floats in LLVM and rustc, you break the parts of libcore that deal with floats.
• Fixes #26449. A pull request to support libcore without floats, closed without merge. A version of this patch is provided by rust-barebones-kernel, and this patch is frequently recommended on #rust-osdev.
• Are floats really dependency free?. Briefly discusses the fact that libcore depends on floating point.
• Pre-RFC: Dealing with broken floating point. This is a more general discussion of floating point issues, but at one point it proposes #[cfg(float_is_broken)]. Not sure how relevant this is.

Datum 1: Some otherwise reasonable processors do not support floating point

There's always been a market for embedded processors without an FPU. For the most part, these aren't pathologically weird processors. The standard ARM toolchain supports --fpu=none. Many of the older and/or lower-end ARM chips lack FPUs. For example, the FPU is optional on the Cortex-M4.

Now, I concur (enthusiastically) that not all embedded processors are suitable for Rust. In particular, there are processors where the smallest integer types are u32 and i32, making sizeof(char) == sizeof(uint32_t) == 1 in C, and where uint8_t literally does not exist. There were once quite a few CPUs with 36-bit words. I agree that all these CPUs are all fundamentally unsuitable for Rust, because Rust makes the simplifying decision that the basic integer types are 8, 16, 32 and 64 bits wide, to the immense relief of everybody who programs in Rust.

But CPUs without floating point are a lot more common than CPUs with weird-sized bytes. And the combination of rustc and libcore is an otherwise terrific toolchain for writing low-level code for this family of architecture.

Datum 2: Linux (and many other kernels) forbid floating point to speed up syscalls and interrupts

Another pattern comes up very often:

1. Everybody likes CPUs with a lot of floating point registers, and even a lot of vector floating point registers.
2. Saving all those floating point registers during a syscall or hardware interrupt can be very expensive. You need to save all the registers to switch tasks, of course, but what if you just want to call write or another common syscall?
3. It's entirely possible to write large amounts of kernel code without needing floating point.

These constraints point towards an obvious optimization: If you forbid the use of floating point registers in kernel space, you can handle syscalls and interrupts without having to save the floating point state. This allows you to avoid calling epic instructions like FXSAVE every time you enter kernel space. Yup, FXSAVE stores 512 bytes of data.

Because of these considerations, Linux normally avoids floating point in kernel space. But ARM developers trying to speed up task switching may also do something similar. And this is a very practical issue for people who want to write Linux kernel modules in Rust.

(Note that this also means that LLVM can't use SSE2 instructions for optimizing copies, either! So it's not just a matter of avoiding f32 and f64; you also need to configure your compiler correctly. This has consequences for how we solve this problem, below.)

Possible solutions

Given this background, I'd argue that "libcore without floats" is a fairly well-defined and principled concept, and not just, for example, a rare pathological configuration to support one broken vendor.

There are several different ways that this might be implemented:

1. Make it possible to disable f32 and f64 when building libcore. This avoids tripping over places where the ABI mandates the use of SSE2 registers for floating point, as in Compiling libcore without SSE leads to LLVM ERROR: SSE register return with SSE disabled rust#26449. The rust-barebones-kernel libcore_nofp.patch shows that this is trivially easy to do.
2. Move f32 and f64 support out of libcore and into a higher-level crate. I don't have a good feel for the tradeoffs here—perhaps it would be good to avoid crate proliferation—but this is one possible workaround.
3. Require support for soft floats in the LLVM & rustc toolchain, even when the platform ABI mandates the use of SSE2 registers. But this is fragile and cumbersome, because it requires maintaining a (custom?) float ABI on platforms even where none exists. And this is currently broken even for x86_64 (Compiling libcore without SSE leads to LLVM ERROR: SSE register return with SSE disabled rust#26449 again), so it seems like this approach is susceptible to bit rot.
4. Compile libcore with floats and then try to remove them again with LTO. This is hackish, and it requires the developer to leave SSE2 enabled at compilation time, which may allow SSE2-based optimizations to slip in even where f32 and f64 are never mentioned, which will subtly corrupt memory during syscalls and interrupts.
5. Other approaches? I can't think of any, but I'm sure they exist.

What I'd like to see is a situation where people can build things like Linux kernel modules, pure-Rust kernels and (hypothetically) Cortex-M4 (etc.) code without needing to patch libcore. These all seem like great Rust use cases, and easily disabling floating point is (in several cases) the only missing piece.

[steveklabnik]

/cc rust-lang/rust#27701

[hanna-kruppe]

[Option 4] Compile libcore with floats and then try to remove them again with LTO. This is hackish, and it requires the developer to leave SSE2 enabled at compilation time, which may allow SSE2-based optimizations to slip in even where f32 and f64 are never mentioned, which will subtly corrupt memory during syscalls and interrupts.

Since LLVM will implement small-ish memcpys by going through XMM registers, this is bound to happen. For example: [u64; 2] copies in release mode. So this option is right out.

[alexcrichton]

I think there's actually a bit of a matrix here which can be helpful when thinking about this, we've got the two vectors of "libcore explicitly uses floating point" and "llvm codegens using floating point". Looking at the possibilities here:

• both turned on -- makes sense, it's what we do today.
• libcore on, llvm off -- looks like this is Compiling libcore without SSE leads to LLVM ERROR: SSE register return with SSE disabled rust#26449, just yields an LLVM error (which kinda makes sense)
• libcore off, llvm on -- this is isn't enough to cover the kernel use case unfortunately as @rkruppe mentioned. Although XMM registers aren't explicitly used LLVM will optimize patterns into using them.
• both turned off -- this is what kernels will need to do (as you've discovered)

With this in mind, I think it may be better to frame this around "disabling floating point support in generated code" rather than specifically omitting it from libcore itself. For example if we look at the above matrix, if LLVM is allowed to use floating point registers, then there's no reason to omit the support from libcore anyway (modulo the fmod issue, which I think is somewhat orthogonal to the usability in kernels).

As a result, this may lead itself quite nicely to a non-invasive implementation. For example on intel processors there may be something like #[cfg(target_feature = "sse2")] which we could use to gate the emission of f32/f64 trait implementations in libcore. To me this makes more sense than "pass a semi-arbitrary cfg flag to libcore and also disable some codegen". I would personally be more amenable to a patch like this to libcore, and note that this also naturally extends itself well I believe to "libcore supports floats if the target does" so weird architectures may be covered by this as well.

[emk]

@rkruppe I agree completely. If we don't want SSE2 instructions, we should tell LLVM not to generate them. Generating them and then trying to remove them will obviously fail.

@alexcrichton Thank you for clarifying the issues! If I understand it, you're proposing two things here:

1. Floats should always be included in libcore if the target supports them, and excluded if it doesn't.
2. This could be implemented by using conditional declarations like #[cfg(target_feature = "sse2")] in libcore.

Am I understanding you correctly? If so, I agree that (1) sounds like a perfectly plausible way to address these issues. But if you intended (2) as a literal proposal (and not just an abstract sketch of an implementation), then I'm not convinced it's the right way to go.

The problem with writing something like #[cfg(target_feature = "sse2")] is that libcore would need to know about every possible platform, and you'd quickly wind up with something like:

#[cfg(or(target_feature = "sse2", target_feature = "x87", target_feature = "neon",
         target_feature = "vfp", target_feature="vfp4", target_feature = "soft_float"))]

...just to cover the Intel and ARM architectures. And depending on how you implemented it, that conditional might have to appear multiple times in libcore. This seems like it would be both ugly and fragile.

Some possible alternatives might be:

 #[cfg(target_feature = "float"))]

…or:

 #[cfg(target_float))]

The advantage of these approaches is that libcore wouldn't need to contain long lists of target-specific features, and the decision-making process could be moved closer to librustc_back/target, which is in charge of other target-specific properties.

Logically, this information feels like it would be either:

1. A TargetOption, or
2. Something that librustc_back/target could infer from Target and TargetOption, using code that knows how to interpret features like sse2 and neon.

I'd guess that (1) is fairly easy to implement, and it would work well with the target *.json files. (2) would require adding new Rust code for each architecture, to interpret features correctly. Either would probably work.

But in the end, I'd be happy to implement just about any of these approaches—whatever works best for you. Like I said, my goal here is to provide a long-term roadmap for safely writing things like kernel modules using libcore, and I'm happy with anything that gets us there. :-)

[alexcrichton]

@emk you're understanding is spot on, that's precisely what I was thinking. I'd be fine extending the compiler to have a higher level notion of "floating point support" and disabling that means something different on every platform, and adding a particular #[cfg] for that seems fine to me!

Some prior art here could be the recent addition of the target_vendor cfg gate. It's both feature gated (e.g. not available on stable by default) but defined in json files as well.

[emk]

Great, thank you for the pointer to target_vendor.

Let me sketch out a design to see if I'm in the right ballpark.

Right now, TargetOptions has a bunch of fields that control specific kinds of code generation features, including disable_redzone, eliminate_frame_pointer, is_like_osx, no_compiler_rt, no_default_libraries and allow_asm.

We could add a has_floating_point to TargetOptions, and a #[cfg(target_has_floating_point)] option behind a feature gate. We could also use better names if anybody wants to propose them. :-) This #[cfg] could be used to conditionalize f32 and f64 in core.

This way, we could define a kernel-safe x86 target using something like:

    "features": "-mmx,-sse,-sse2,-sse3,-ssse3,-sse4.1,-sse4.2,-3dnow,-3dnowa,-avx,-avx2",
    "has-floating-point": false,
    "disable-redzone": true,

I think that this would actually be a fairly small, clean patch. (Alternatively, we could try something more ambitious, where has_floating_point automatically implied the corresponding features list, but that would probably require adding another field named something like features_to_disable_floating_point to TargetOptions.)

Would this design work as is? If not, how could I improve it? If we can come up with a basically simple and satisfactory design, I'd be happy to try to implement it. Thank you for your feedback!

[alexcrichton]

@emk yeah that all sounds good to me, I'd be fine if disabling floats just implied all the necessary features to pass down to LLVM so they didn't have to be repeated as well.

[emk]

@alexcrichton Thank you for the feedback!

For a first pass, I'll try to implement has_floating_point in TargetOptions. If we want that to automatically disable the corresponding features, though, we'd probably still need to specify what that means in the target file, at least in the general case:

    "features": "...",
    "disable-floating-point-features": "-mmx,-sse,-sse2,-sse3,-ssse3,-sse4.1,-sse4.2,-3dnow,-3dnowa,-avx,-avx2",
    "has-floating-point": false,
    "disable-redzone": true,

Above, we disable floating point, and then we need explain what that means, so that the compiler can do it for us.

I'm not sure that's really an improvement over:

    "features": "-mmx,-sse,-sse2,-sse3,-ssse3,-sse4.1,-sse4.2,-3dnow,-3dnowa,-avx,-avx2",
    "has-floating-point": false,
    "disable-redzone": true,

Can anybody think of a better design? I'm definitely open to suggestions here, and I'm sure I don't see all the use cases.

And thank you again for your help refining this design!

[alexcrichton]

Hm yeah that's a good point I guess, the features being passed down do probably need to be generic. It's a little unfortunate that you can still construct an invalid target specification by disabling floating point and not disabling the features, but I guess that's not necessarily the end of the world.

[emk]

OK, I've planned out a block of time to work on this week (hopefully by midweek-ish, if all goes well).

[steveklabnik]

Ran into this again today :) @emk did you get a chance to work on it at all?

[emk]

Not yet! I'm bounding between two different (free-time) Rust projects right
now, and this affects the other one. You're welcome to steal this out from
under me if you wish, or you can prod me to go ahead and finish it up as
soon as possible. :-)

Le sam. 12 déc. 2015 à 12:02, Steve Klabnik notifications@github.com a
écrit :

Ran into this again today :) @emk https://github.com/emk did you get a
chance to work on it at all?

—
Reply to this email directly or view it on GitHub
#1364 (comment).

[steveklabnik]

Okay :) It's not mega urgent for me, either, so I might or might not :)