Reintroduce support for negative epoch_millis in sub-second granularity #1991
Description
Context and Problem
Elasticsearch 7 introduced a regression to the date field type. Negative epoch_milli timestamps that have second fractions lead to a parse exception. This was not the case in previous Elasticsearch versions. This pull request describes the issue quite well and shows a possible solution approach: elastic/elasticsearch#80208.
The full support for arbitrary negative epoch_millis could be a reason for users of Elasticsearch 6 to migrate to OpenSearch instead of Elasticsearch 7.
Proposed solution
The solution proposed in the aforementioned pull request is nice because it works around the underlying limitation, keeping the API stable.
Alternatives to work around the regression
Elastic's suggestion for dealing with the regression is to refrain from using negative epoch_millis and instead encode these dates using the ISO 8601 string representation. This forces changes to client code, which is not always possible.
Additional context
Here a sequence of requests that illustrate the issue:
Create the index:
curl -X PUT https://localhost:9200/test -u 'admin:admin' --insecure -H "Content-Type: application/json" -d '{
"mappings": {
"properties" : {
"date" : {
"type" : "date",
"format" : "epoch_millis"
}
}
}
}'
{"acknowledged":true,"shards_acknowledged":true,"index":"test"}
It is possible to index a document with negative epoch_millis as long as it has no second fractions:
curl -X PUT https://localhost:9200/test -u 'admin:admin' --insecure -H "Content-Type: application/json" -d '{ "date" : -1643201736000 }'
{"_index":"test","_type":"_doc","_id":"1","_version":1,"result":"created","_shards":"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}
But it's not possible to index a document with negative epoch_millis with second fractions:
curl -X PUT https://localhost:9200/test/_doc/2 -u 'admin:admin' --insecure -H "Content-Type: application/json" -d '{ "date" : -1643201736001 } '
{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse field [date] of type [date] in document with id '2'. Preview of field's value: '-1643201736001'"}],"type":"mapper_parsing_exception","reason":"failed to parse field [date] of type [date] in document with id '2'. Preview of field's value: '-1643201736001'","caused_by":{"type":"illegal_argument_exception","reason":"failed to parse date field [-1643201736001] with format [epoch_millis]","caused_by":{"type":"date_time_parse_exception","reason":"Failed to parse with all enclosed parsers"}}},"status":400}