Skip to content

data_nanos based search_after query with custom timestamp format causes exception #52424

Closed
@kertal

Description

@kertal

Elasticsearch version (bin/elasticsearch --version):
Master

JVM version (java -version):

OS version (uname -a if on a Unix-like system):

Darwin Kernel Version 18.7.0

Description of the problem including expected versus actual behavior:

A query using search_after based on an index using a date_nanos with a custom timestamp format throws an exception

Steps to reproduce:

  1. Create custom mapping
PUT date_nanos_custom_timestamp
{
    "mappings": {
      "properties": {
        "timestamp": {
          "format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
          "type": "date_nanos"
        }
      }
    },
    "settings": {
      "index": {
        "number_of_replicas": "1",
        "number_of_shards": "1"
      }
    }
}
  1. Add same sample data
PUT date_nanos_custom_timestamp/_doc/1
{
  "timestamp": "2019-10-21 00:30:04.828740"
}

PUT date_nanos_custom_timestamp/_doc/2
{
  "timestamp": "2019-10-21 08:30:04.828733"
}
  1. Search
POST date_nanos_custom_timestamp/_search
{
  "size": 5,
  "search_after": [
    "1571617804828740000",
    1
  ],
  "sort": [
    {
      "timestamp": {
        "order": "asc",
        "unmapped_type": "boolean"
      }
    },
    {
      "_doc": {
        "order": "asc",
        "unmapped_type": "boolean"
      }
    }
  ]
  }
}

Now you're getting an exception:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "parse_exception",
        "reason" : "failed to parse date field [1571617804828740000] with format [yyyy-MM-dd HH:mm:ss.SSSSSS]: [failed to parse date field [1571617804828740000] with format [yyyy-MM-dd HH:mm:ss.SSSSSS]]"
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "date_nanos_custom_timestamp",
        "node" : "059iRHeFT-ObiKskYJ_29w",
        "reason" : {
          "type" : "parse_exception",
          "reason" : "failed to parse date field [1571617804828740000] with format [yyyy-MM-dd HH:mm:ss.SSSSSS]: [failed to parse date field [1571617804828740000] with format [yyyy-MM-dd HH:mm:ss.SSSSSS]]",
          "caused_by" : {
            "type" : "illegal_argument_exception",
            "reason" : "failed to parse date field [1571617804828740000] with format [yyyy-MM-dd HH:mm:ss.SSSSSS]",
            "caused_by" : {
              "type" : "date_time_parse_exception",
              "reason" : "Text '1571617804828740000' could not be parsed at index 0"
            }
          }
        }
      }
    ]
  },
  "status" : 400
}

This worked a while ago, I've created a functional test in Kibana

elastic/kibana#54089

But had to skip it, since it caused the 400

elastic/kibana#56178

This works when providing the sort params as number

 "search_after": [
    1571617804828740000,
    1
  ],

But that's no a solution due to Browser limitation with BigInt numbers

Metadata

Metadata

Assignees

Labels

:Search/SearchSearch-related issues that do not fall into other categoriesTeam:SearchMeta label for search team

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions