EPSILON is a bad error margin and should not be recommended [float_cmp]

[CAD97]

[f32|f64]::EPSILON are the machine epsilon of the type, or (as stated in the Rust docs), the distance between 1.0 and the next representable floating point number.

The page linked in the more info specifically says that abs( a - b ) < epsilon is wrong for any value of espilon. However, it's especially egregious with f__::EPSILON, because for floating point numbers outside the range -2..=2, floating point numbers cannot be f__::EPSILON close, so abs( a - b ) < f__::EPSILON is actually equivalent to a strict equality check.

There isn't a generally applicable solution to recommend. The most thorough resource I've found suggests comparison in ULPs for testing against a non-zero number, and testing against a fixed epsilon (but one bigger than f__::EPSILON.

At the least, we (and probably std) shouldn't be recommending comparing against f__::EPSILON, as it's basically as poor as bitwise equality and gives a false sense of handling the problem, when it isn't really handled.

[llogiq]

When I was still writing float-heavy code, I always had a function to calculate a suitable epsilon-value based on the values to compare and the number of mantissa bits that I expected to be equal.

[camsteffen]

So then should we lint against abs(a - b) < f__::EPSILON?

[camsteffen]

This also has implications for float_equality_without_abs.

[CAD97]

So then should we lint against abs(a - b) < f__::EPSILON?

I'm not completely certain. In cases where a and b are expected to be "small" (for some value of "small" smaller than 1), this might be a reasonable comparison epsilon, because ULPs for small enough floats are really small, and the machine epsilon might be a reasonable bound for "small enough". At the same time, the article I linked recommends some small multiple of the machine epsilon for these cases.

That said, I do think that abs(a - b) < f__::EPSILON most likely came about by an unfortunately misguided attempt to address the "imprecise floats problem." As such, I think a lint against specifically < f__::EPSILON is fair, so long as it doesn't fire for small multiples of the machine epsilon, as that shows that some amount of thought went into picking the comparison epsilon.

To be extra pedantic, a pedantic lint against abs(a - b) < ANY_FLOAT_CONST where a and b are function arguments could be argued for, pointing out how a fixed epsilon is subtly wrong for arbitrary floating point due to its varrying precision.

I don't have any strong opinions on linting float comparisons, though, beyond not recommending comparison against machine epsilon.

The old classic "What Every Computer Scientist Should Know About Floating-Point Arithmetic" says that when a real number is rounded to the closest floating point number it's relative error is bounded by machine epsilon. Note that is relative error not absolute.

So based on that I would expect the correct comparison to be abs( (a - b)/a ) <= epsilon for a != 0. When a is zero I guess you could use fxx::MIN_POSITIVE.

Oh, it looks like fxx::MIN_POSITIVE is the smallest positive normal value. So that wouldn't be right.

[trentj]

This bad suggestion came up on URLO.