Miri floating point NaN conversion issue

[jodysankey]

I tried this code:

fn main() {
    let nan1 = f64::from_bits(0x7FF0_0001_0000_0001u64);
    let nan2 = f64::from_bits(0x7FF0_0000_0000_0001u64);

    assert!(nan1.is_nan());
    assert!(nan2.is_nan());

    let nan1_32 = nan1 as f32;
    let nan2_32 = nan2 as f32;

    assert!(nan1_32.is_nan());
    assert!(nan2_32.is_nan());

This runs correctly (i.e. doesn't assert) when run with rust nightly, but fails at assert!(nan2_32.is_nan()); when run with miri on nightly (2020-02-20).

It seems like miri just truncated the f64 to get the f32 and therefore converted the nan2 to a floating point zero instead of a floating point NaN

[RalfJung]

(Miri-the-tool has its own bugtracker at https://github.com/rust-lang/miri/issues, where I see things earlier than when they are burried here.)

Cc @eddyb, this could also be an apfloat issue as that's what Miri uses to implement casts.
I don't think we are doing any truncation here. Would be interesting to see the bit patterns involved.

[RalfJung]

to_bits for nan2_32 returns 0x7f800000. So, this doesn't look like just a truncated nan2 to me, it looks like apfloat did something but that something is not correct (or at least, not nan-preserving -- I am not sure if that is a guarantee).

[eddyb]

cc @hanna-kruppe

[jodysankey]

Sorry, I should have said truncated the fraction there. The 0x7f8 is the floating point exponent being set to all ones, the 0x00000 (plus the empty bottom 3 bits in the 8) is a truncation of the original 0x1 fraction in the f64. By removing all the non-zero bits in the fraction it converted the NaN to an infinity.

For reference to_bits for nan1_32 returns 0x7f800008 where the 0x00008 (plus the empty bottom 3 bits in the 8) is a truncation of the original 0x0000100000001 fraction from 52 to 23 bits

Note that the normal Rust compiler seems to always set the MSB in the fraction of a NaN during its conversion to avoid this problem. There nan1_32 is 0x7fc00008 and nan2_32 is 0x7fc00000

[RalfJung]

There nan1_32 is 0x7fc00008 and nan2_32 is 0x7fc00000

Assuming that f64-to-f32 on NaN has fully defined semantics, we should probably have a test asserting these exact bit patterns for Miri.

But what you wrote makes it sound even more like the bug is in apfloat, and not in the Miri code that feeds data to/from apfloat. FWIW, the float casting code is here.

[jodysankey]

Yes, it does sound to me like an apfloat issue rather than something unique in Miri. Reading that casting method I can't see anything that is special casing the fraction truncation for NaNs, only for an 80bit extended precision type that has different NaN semantics.

Sorry, I've not dealt with the bug workflow on Rust before - is it possible to move this bug to apfloat? Would you like me to create a separate bug on apfloat?

[RalfJung]

apfloat does not have a separate issue tracker, no need to move anything.

However, if I understood @eddyb correctly then apfloat is a direct Rust port of LLVM's apfloat -- so likely they have the same bug, and a fix would have to happen upstream as well.

[hanna-kruppe]

Yes, rustc_apfloat is a port of LLVM's APFloat, but LLVM folds this cast to 0x7FF0_0000 (also in older versions, back to before @eddyb started the Rust port). Note that the MSB of the significand is set, so this is indeed a NaN. It seems LLVM does not have the same bug. Something strange is up.

[RalfJung]

LLVM folds this cast to 0x7FF0_0000

Hm, that's still not the same as the 0x7fc00000 that we see at run-time though?

[RalfJung]

I think the LLVM C++ code doing the same cast is here.

[hanna-kruppe]

LLVM folds this cast to 0x7FF0_0000

Hm, that's still not the same as the 0x7fc00000 that we see at run-time though?

Sure, but that's intentional. Optimizations need to preserve whether something is a NaN but (by default) the payload bits, quiet/signaling distinction and sign bit are not generally preserved or guaranteed to match between compile time evaluation and run time evaluation.