Transforms fails due to invalid year field when using a particular date

[djaswal]

Elasticsearch version (bin/elasticsearch --version): 7.9.1

Plugins installed: [] - None

JVM version (java -version): 1.8.0_222-ea

OS version (uname -a if on a Unix-like system): Ubuntu 18.04

Description of the problem including expected versus actual behavior:
Transform should succeed for any dates instead of failing with the following message:

Preview of field's value: '-7948800000']; nested: DateTimeException[Invalid value for Year (valid values -999999999 - 999999999): -7948800000];

Steps to reproduce:

Please include a minimal but complete recreation of the problem,
including (e.g.) index creation, mappings, settings, query etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.

1. Create index

PUT test_date_transform
  {
  "mappings": {
    "properties": {
      "date":    { "type": "date" }
    }
  }
  }

2. Add the specific datetime to the index

POST test_date_transform/_doc
{
  "date": "1969-10-01T00:00:00",
  "value": "test2"
}

3. Create the transform

PUT _transform/testtransform
{
  "source": {
    "index": [
      "test*"
    ]
  },
  "pivot": {
    "group_by": {
      "date": {
        "date_histogram": {
          "field": "date",
          "calendar_interval": "1m"
        }
      }
    },
    "aggregations": {
      "value.keyword.cardinality": {
        "cardinality": {
          "field": "value.keyword"
        }
      }
    }
  },
  "dest": {
    "index": "transform_destination"
  }
}

4. Start the transform and it will fail.

Provide logs (if relevant):
Failed to index documents into destination index due to permanent error: [BulkIndexingException[Bulk index experienced [1] failures and at least 1 irrecoverable [TransformException[Destination index mappings are incompatible with the transform configuration.]; nested: MapperParsingException[failed to parse field [date] of type [date] in document with id '_2vj-jAZwoUNV8WjDqeWhPQAAAAAAAAA'. Preview of field's value: '-7948800000']; nested: DateTimeException[Invalid value for Year (valid values -999999999 - 999999999): -7948800000];; MapperParsingException[failed to parse field [date] of type [date] in document with id '_2vj-jAZwoUNV8WjDqeWhPQAAAAAAAAA'. Preview of field's value: '-7948800000']; nested: DateTimeException[Invalid value for Year (valid values -999999999 - 999999999): -7948800000];; java.time.DateTimeException: Invalid value for Year (valid values -999999999 - 999999999): -7948800000]. Other failures: ]; nested: TransformException[Destination index mappings are incompatible with the transform configuration.]; nested: MapperParsingException[failed to parse field [date] of type [date] in document with id '_2vj-jAZwoUNV8WjDqeWhPQAAAAAAAAA'. Preview of field's value: '-7948800000']; nested: DateTimeException[Invalid value for Year (valid values -999999999 - 999999999): -7948800000];; TransformException[Destination index mappings are incompatible with the transform configuration.]; nested: MapperParsingException[failed to parse field [date] of type [date] in document with id '_2vj-jAZwoUNV8WjDqeWhPQAAAAAAAAA'. Preview of field's value: '-7948800000']; nested: DateTimeException[Invalid value for Year (valid values -999999999 - 999999999): -7948800000];; MapperParsingException[failed to parse field [date] of type [date] in document with id '_2vj-jAZwoUNV8WjDqeWhPQAAAAAAAAA'. Preview of field's value: '-7948800000']; nested: DateTimeException[Invalid value for Year (valid values -999999999 - 999999999): -7948800000];; java.time.DateTimeException: Invalid value for Year (valid values -999999999 - 999999999): -7948800000]

[elasticmachine]

Pinging @elastic/ml-core (:ml/Transform)

[benwtrent]

@djaswal I think this bug has been fixed in some recent work here: #65952

This is part of the 7.11 branch, which means it will hopefully be released in 7.11.

[hendrikmuhs]

I confirmed #65584 (backport #65952) fixes this problem:

GET transform_destination/_search returns:

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "transform_destination",
        "_id" : "_2vj-jAZwoUNV8WjDqeWhPQAAAAAAAAA",
        "_score" : 1.0,
        "_source" : {
          "date" : "1969-10-01T00:00:00.000Z",
          "value" : {
            "keyword" : {
              "cardinality" : 1
            }
          }
        }
      }
    ]
  }
}

Note how date is written, in <7.11 it is writting as time since epoch, that's why it failed, >=7.11 it is pushed as a proper ISO string.

@djaswal Thanks for the report, the fix will become available with 7.11.