-
Notifications
You must be signed in to change notification settings - Fork 187
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Implement ties-to-even for Big[U]Int::to_{f32,f64}
BigUInt::to_{f32,f64} uses only the highest 64 bits as the mantissa, discarding the rest, which gives an incorrect result when later converted to float using IEEE-754's default nearest-ties-to-even rounding. Fixed to use correct round-to-odd for taking the mantissa, which means setting the LSB to 1 if any of the lower (i.e. rounded away) bits are 1. This also fixes the problem for BigInt since it really just uses BigUInt under the hood.
- Loading branch information
1 parent
6f2b8e0
commit 7433d22
Showing
3 changed files
with
50 additions
and
10 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters