BUG: float conversion in read_csv is inaccurate for precise input

[mdmueller]

This is coming from #2566, which noted that the xstrtod() method used by read_csv doesn't agree with standard numpy float conversion. I see that the priority is speed of parsing over complete accuracy (within 0.5 units in the last place, or ULP), but there are two issues here that (at least to me) actually appear to be buggy:

1. Low-precision values (i.e. less than about 15 significant figures) should be guaranteed to be within 0.5 ULP of the correct result, or at least within 1 ULP. However, I have found cases in which xstrtod() is off by more than 1 ULP, although these don't come up often.
2. High-precision values of course can't be guaranteed to be within 0.5 ULP without a costly correction loop as in the ordinary strtod(), but the error in conversion increases linearly as the number of supplied significant figures increases. With 30 significant figures, the error in conversion can potentially be over 7 ULP.

Here is an IPython notebook analyzing the accuracy of xstrtod(). I think there are two problems here: xstrtod() keeps reading digits after the 17th, none of which should matter for conversion, and the scaling step at the end produces a compounded error by repeatedly multiplying/dividing by powers of 10. I have a solution for AstroPy that seems to fix these issues, so I can open a PR if it's agreed that xstrtod() should be changed.

[jreback]

how about an option for this (default is turned off), may float_precision=None|high ?

[mdmueller]

That could work--our decision in AstroPy was also to use a parameter named use_fast_converter, although we have it defaulting to False. Then should the updated xstrtod() be used if float_precision='high', or just strtod() (which is guaranteed to be within 0.5 ULP)?

[jreback]

I would like to leave the existing code in-place. No objection to adding new options like this though. I don't think the intermediate (using strtod()) is that useful, while a higher-precision prob is. The tradeoff of course is speed/accuracy.

[mdickinson]

Just to add another voice: I also ran into this issue recently. We were serialising data to disk via DataFrame.to_csv and then re-reading with read_csv, and were surprised to discover that the re-read data didn't exactly match the original. While the errors are small, and not an issue for calculation, it's important to us that the serialisation round-trips correctly. So a float_precision='high' option would be useful.

[jreback]

ok @amras1 we'll put this on the table for a new option.

[mdmueller]

Sounds good--I have the altered xstrtod() ready, so let me know if I should open a PR. However, the new xstrtod() isn't guaranteed to fix the round-trip issue @mdickinson noticed; strtod() is probably the best solution for that case.

[jreback]

I suppose you could have 3 levels for float_precision=None|high|round_trip or something where you either: use current xstrtod, new one, strtod