Tracking Issue for `extern "rust-cold"` ABI (feature `rust_cold_cc`)

[scottmcm]

This is a tracking issue for the experimental extern "rust-cold" ABI.
The feature gate for the issue is #![feature(rust_cold_cc)].

This allows emitting functions with coldcc in LLVM and CallConv::Cold in cranelift. These are more aggressive than the stable #[cold], as they can use a different calling convention (and thus have incompatible fns), not just affect branch weights -- for example, they might require more/slower code in the callee in exchange for smaller call sites.

About tracking issues

Tracking issues are used to record the overall progress of implementation.
They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions.
A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature.
Instead, open a dedicated issue for the specific matter and add the relevant feature gate label.

Steps

• Implement
• Adjust documentation (see instructions on rustc-dev-guide)
• Stabilization PR (see instructions on rustc-dev-guide)

Unresolved Questions

• How can we make using this compatible with mixing codegen backends -- particularly in the context of using an LLVM-compiled standard library with a cranelift-compiled binary or tests cc Use preserve_mostcc for extern "rust-cold" #115260 (comment)
• Should this be coldcc, preservemost_cc, or something else in LLVM?
  • clang offers clang::preserve_most but nothing named "cold", FWIW: https://clang.llvm.org/docs/AttributeReference.html#calling-conventions
  • As a Rust-* convention we're not promising anything, so can always change the details later.

Implementation history

• Add support for emitting functions with coldcc to LLVM #97512
• Don't use coldcc Use preserve_mostcc for extern "rust-cold" #115260
• compiler: Accept "improper" ctypes in extern "rust-cold" fn #130667 - Suppresses extern "C" improper ctypes lint, since this is a Rust ABI

[reinerp]

I'm very excited about this, thank you for your work on this!

Couple comments:

• this seems to be broken for 16-byte returns on x86-64. Example: https://play.rust-lang.org/?version=nightly&mode=debug&edition=2021&gist=e484bf958a2e5b421de9e5a52f736723. Looking at the code for foo in this example, it seems to be trying to return the result in rax and rdx, but then it calls pop rdx which clobbers the return value.
• coldcc is suitable for "never runs" things like panics. For "runs rarely (amortized)" things, LLVM's preserve_mostcc seems like it is often a better fit. (A primary difference is that coldcc saves all XMM registers, whereas preserve_mostcc does not.) It has the added advantage of being exposed to C++ in Clang, and seems to be used in some popular codebases (V8, Folly), so it might be better exercised/tested than the coldcc convention. Would it be possible to have a "rust-cool" variant too, which uses preserve_mostcc?