Skip to content

Fix precision loss in compareDoubleInt/compareDoubleUint for values outside float64 safe integer range#1328

Open
XananasX7 wants to merge 2 commits into
google:masterfrom
XananasX7:fix/compareDoubleInt-precision
Open

Fix precision loss in compareDoubleInt/compareDoubleUint for values outside float64 safe integer range#1328
XananasX7 wants to merge 2 commits into
google:masterfrom
XananasX7:fix/compareDoubleInt-precision

Conversation

@XananasX7
Copy link
Copy Markdown

Summary

Fix precision loss in compareDoubleInt and compareDoubleUint when comparing double with int/uint values outside the safe float64 integer range (> 2^53).

Problem

The current implementation casts Int (int64) or Uint (uint64) to float64 before comparison:

func compareDoubleInt(d Double, i Int) Int {
    // ...bounds checks...
    return compareDouble(d, Double(i))  // Double(i) = float64(i): loses precision!
}

IEEE 754 float64 has a 53-bit mantissa. Any integer with absolute value greater than 2^53 = 9007199254740992 cannot be represented exactly as float64. This means two distinct int64 values can map to the same float64, causing the comparison to incorrectly return equal.

Demonstrated bug

// int(9007199254740993) == double(9007199254740992.0) evaluates to TRUE
// but these are different numbers — off by 1
result := compareDoubleInt(Double(9007199254740992), Int(9007199254740993))
// result == 0 (equal) — WRONG

Other affected values:

  • int(100000000000000001) == double(1e17)true (should be false)
  • int(1000000000000000001) == double(1e18)true (should be false)

Impact

CEL is used as the expression engine in GCP IAM Conditions, Firebase Security Rules, and Kubernetes admission webhooks. A policy rule that compares an integer field to a double literal near these boundary values can produce incorrect authorization decisions.

Fix

Use math/big.Float for exact comparison, eliminating the int→float64 precision loss:

func compareDoubleInt(d Double, i Int) Int {
    if d < math.MinInt64 { return IntNegOne }
    if d > math.MaxInt64 { return IntOne }
    bf := new(big.Float).SetFloat64(float64(d))
    bi := new(big.Float).SetInt64(int64(i))
    return Int(bf.Cmp(bi))
}

The same fix is applied to compareDoubleUint.

The fast-path bounds checks (< MinInt64, > MaxInt64) remain unchanged and avoid the big.Float allocation for values clearly out of range. The big.Float path is only reached for values in the representable int64 range where precision matters.

Testing

All existing comparison tests continue to pass. The previously incorrect cases now return the correct result:

int(9007199254740993) == double(9007199254740992.0) → false ✅
int(100000000000000001) == double(1e17)             → false ✅
int(1000000000000000001) == double(1e18)            → false ✅

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant