Description
openedon Jul 2, 2019
The problem
JavaScript (in contrast to Java which Elasticsearch uses) does not support 64 bit integer values. Using a number in JavaScript results in the Number
type which itself is a 64bit IEEE floating point number, thus leaving us a 52 bit mantisse to store integer values without rounding errors. Everything above what fits into 52 bit will have rounding errors.
You can see that in the following screenshot when entering a number that doesn't fit into 52 bit anymore, JavaScript cannot represent it correctly anymore. So even though I wrote two different numbers (1st and 3rd line) they both are the same number for JavaScript.
That means every integer number above 9007199254740991 (Number.MAX_SAFE_INTEGER
) or below -9007199254740991 (Number.MIN_SAFE_INTEGER
) will be represented with rounding errors in Kibana, even if Elasticsearch can handle it correctly and return it correctly.
This problem is intrinsic to JavaScript and you should be careful when trying to debug something like that, since even the Chrome Dev Tools when using the "Preview" tab on a response will round those values, and only the "Response" tab will show the result as it really was returned from the Server. This is though only true for JSON responses. For APIs returning ndjson (new line separated JSON) as our internal search endpoints, values will be rounded in the "Preview" and in the "Response" tab.
Effect in Kibana
Thus fields in Elasticsearch that can contain numbers outside that safe JavaScript range (e.g. long
, date_nanos
, unsigned_long
) are prone to that rounding error in several parts of Kibana. This section should summarize what might not work in Kibana or how it behaves in Kibana.
Dev Tools > Console
✔️ works (with minor limitations)
You will see no rounding errors in the Console's requests or response from 6.5 onwards, since this will never be "interpreted" as an object with potential numbers, but Kibana simply output the bytestream from the response into that panel. (#23685 introduced this)
In 6.4 and earlier versions of Kibana, the same rounding error problem will occur in console as in other parts of Kibana.
Limitation: Performing a code reindent in the console is impacted by this issue, as the reindent logic is parsing the JSON content, i.e. the request after reindentation might have rounded values from what the user entered beforehand. See #101391
Discover
❌ several issues
When looking at documents containing a value outside that safe range, it will be shown in Discover with this rounding error, thus not necessarily representing the real value in Elasticsearch. This is true for the primary Discover table AS for the JSON tab (including values inside _source
). Though the table round it "more specific", thus those two values, might still be different from each other, but both might be rounded, see the following example where the value "9174563637958049791" was injected into a long
field in Elasticsearch:
Sorting on a field containing those values will still work as expected (even though the values are not shown correctly), since we let Elasticsearch do the sorting, which doesn't have that issue.
A field of type date_nanos
will actually work and also show the correct nanosecond value, because we don't treat them as numbers, but use Elasticsearch's capabilities to return them as date strings, which we can then print and never lose precision by touching that field as a number.
Prior to 7.13 or when having the advanced setting discover:readFieldsFromSource
switched on, you might even see this rounding issues in Discover on fields of type keyword
or text
if they were inserted into Elasticsearch as a number. Since we print the value returned in _source
and this value will represent the value as it was indexed (meaning if it wasn't put into quotes as a string while indexing, we will neither get it returned as a string). See #38398 for an example.
Visualizations
❌ several issues
Visualizing date_nanos
is safe, since Elasticsearch treats them as milliseconds for aggregations, which are inside the safe range for numbers in JavaScript.
For all fields really containing values outside that safe integer range in JavaScript it actually depends a bit on how they are used in visualizations.
If you are creating buckets using such a field, each document will be in the correct bucket (since that's handled by Elasticsearch), but you might see a couple of weird effects:
- Buckets that due to the rounding error have the same key would be merged into one bucket in some visualizations (this is actually a rather rare case, and you most likely won't have such small buckets that the rounding error actually give them the same key).
- For all those values outside the safe range, you will see the rounded values on axes and tooltips.
- If you visualize such a value e.g. on the y-axis, the values drawn inside the charts are the rounded ones not the correct ones returned from Elasticsearch. You can see this in the following example chart. I indexed 5 documents, and all of them have a different value in the
value
field, but 3 of those documents round to the same number and 2 of them to the other same number, so the chart looks as follows, even though all the values where continuously decreasing from document to document:
KQL
✔️ Everything works from 7.12 onwards
Prior to 7.12 KQL queries were prone to rounding errors. If you wanted to query for large integers outside that safe range in KQL, you'll be affected by the rounding issue, e.g. if you're looking for document were number > 9007199254740993
you'll also retrieve documents that had a value of 9007199254740992 in it, since the value in the query was rounded.
Inspector
❌ several issues
The Inspector in Kibana, that can be used to investigate request/responses is affected by that issue, meaning responses shown there, might already contain rounded values and not fully reflect what Elasticsearch actually returned.
The future
There actually is a stage 3 4 proposal (the most mature level before getting into the standard) for JavaScript (and implemented in TypeScript since 3.2) that adds a BigInt
datatype into JavaScript (https://github.com/tc39/proposal-bigint). If we would have that, we could potentially fix that problem, though it would still take a lot of effort, since we would need to make sure we're having a custom JSON parser and not let the browser touch any response and potentially lose that precision by parsing a numeric value as a pure Number
. Then we would need to make sure everything from the first custom parsing into BigInt
to visualizations are aware that numeric values could also be BigInt
. This will be a huger undertaking, once JavaScript would support that datatype.