Precision error in ToRawPrecision

[eddydas]

In ToRawPrecision, you have these lines:

f = Math.round(Math.exp((Math.abs(e - p + 1)) * Math.LN10)),
m = String(Math.round(e - p + 1 < 0 ? x * f : x / f));

f is very prone to precision error:
// f = Math.round(Math.exp((Math.abs(20)) * Math.LN10))
// 100000000000001000000

This makes m very inaccurate, at least in our targeted precision of 21 digits.
Result is that numbers formatted using SignificantDigits is not accurate after ~10 figures.

[andyearnshaw]

Yeah, number precision has been a pain since the start. I've looked at different ways of fixing it and done the best I could to get it as stable as possible. Even with the last fix I noticed some issues and this is something I need to write some fresh unit tests for.

I'll have another look at this when I get a bit of time (unless anyone has any ideas!).

[eddydas]

I tried simply getting parseInt('1'+str_repeat('0',
number_of_zeros_to_add)) and it seem to solve the problem.

On Mon, Jul 7, 2014 at 11:44 PM, Andy Earnshaw notifications@github.com
wrote:

Yeah, number precision has been a pain since the start. I've looked at
different ways of fixing it and done the best I could to get it as stable
as possible. Even with the last fix I noticed some issues and this is
something I need to write some fresh unit tests for.

I'll have another look at this when I get a bit of time (unless anyone has
any ideas!).

—
Reply to this email directly or view it on GitHub
#71 (comment).

[andyearnshaw]

Are you only seeing this problem for powers of 10? If that's the case, we might be able to do something along the lines of +('1e' + numberOfZeros).

I thought about adding a special case for powers of 10 before, when you raised a similar issue. I really need to get around to writing those unit tests.

[eddydas]

My solution attempts to fix the value for "f", which can only be powers of 10 only. Once "f" is made accurate, "m" is also accurate (well, at least relatively).

Basically it tries to do 10^Math.abs(e - p + 1). When we use Math.LN10 and exp(), it becomes super inaccurate. So the idea is just to get rid of exp() as much as possible, thus my solution.

[andyearnshaw]

Yeah, it's the floating point precision issue rearing its ugly head again. OK, so I will take a look at it when I have a minute again.

[caridy]

This might be related to #169