GeoIP injected fields no longer referable by subsequent processors as of 6.5.0

[Mpdreamz]

As of 6.5.0 the rename processor complains about fields injected by the geoip processor.

field [leadDeveloper.geoIp.city_name] doesn't exist

Possibly relates to either:

• https://github.com/elastic/elasticsearch/pull/33029/files
• https://github.com/elastic/elasticsearch/pull/33840/files

Given the following pipeline:

PUT http://127.0.0.1:9200/_ingest/pipeline/pipeline-33a33a6c HTTP/1.1
Accept: application/json
Content-Type: application/json
Content-Length: 942
Host: 127.0.0.1:9200

{
  "description": "Index pipeline test",
  "processors": [
    {
      "rename": {
        "field": "lastActivity",
        "target_field": "lastSeen"
      }
    },
    {
      "geoip": {
        "field": "leadDeveloper.ipAddress",
        "target_field": "leadDeveloper.geoIp"
      }
    },
    {
      "rename": {
        "field": "leadDeveloper.geoIp.continent_name",
        "target_field": "leadDeveloper.geoIp.continentName"
      }
    },
    {
      "rename": {
        "field": "leadDeveloper.geoIp.city_name",
        "target_field": "leadDeveloper.geoIp.cityName"
      }
    },
    {
      "rename": {
        "field": "leadDeveloper.geoIp.country_iso_code",
        "target_field": "leadDeveloper.geoIp.countryIsoCode"
      }
    },
    {
      "rename": {
        "field": "leadDeveloper.geoIp.region_name",
        "target_field": "leadDeveloper.geoIp.regionName"
      }
    }
  ]
}

Sending the following on 6.4.0:

PUT http://127.0.0.1:9200/project/doc/nest-fluent-42f26033?pretty=true&error_trace=true&routing=nest-fluent-42f26033&refresh=true&pipeline=pipeline-33a33a6c HTTP/1.1
Accept: application/json
Content-Type: application/json
Content-Length: 451
Host: 127.0.0.1:9200

{
  "curatedTags": [
    {
      "added": "2015-06-06T12:01:02.123",
      "name": "x"
    }
  ],
  "join": "project",
  "lastActivity": "2015-06-06T12:01:02.123",
  "leadDeveloper": {
    "ipAddress": "193.4.250.122",
    "id": 1
  },
  "name": "nest-fluent-42f26033",
  "numberOfContributors": 0,
  "startedOn": "2015-06-06T12:01:02.123",
  "state": "Stable",
  "type": "project",
  "visibility": "Public"
}

returns:

HTTP/1.1 201 Created
Location: /project/doc/nest-fluent-42f26033?routing=nest-fluent-42f26033
Warning: 299 Elasticsearch-6.4.0-595516e "the default number of shards will change from [5] to [1] in 7.0.0; if you wish to continue using the default of [5] shards, you must manage this on the create index request or with an index template" "Mon, 28 Jan 2019 09:53:27 GMT"
content-type: application/json; charset=UTF-8
content-length: 266

{
  "_index" : "project",
  "_type" : "doc",
  "_id" : "nest-fluent-42f26033",
  "_version" : 1,
  "result" : "created",
  "forced_refresh" : true,
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 0,
  "_primary_term" : 1
}

as of 6.5.0 this now returns:

HTTP/1.1 500 Internal Server Error
processor_type: rename
Warning: 299 Elasticsearch-6.5.3-159a78a "the default number of shards will change from [5] to [1] in 7.0.0; if you wish to continue using the default of [5] shards, you must manage this on the create index request or with an index template" "Mon, 28 Jan 2019 09:59:24 GMT"
content-type: application/json; charset=UTF-8
content-length: 7036

{
  "error" : {
    "root_cause" : [
      {
        "type" : "exception",
        "reason" : "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [leadDeveloper.geoIp.city_name] doesn't exist",
        "header" : {
          "processor_type" : "rename"
        },
        "stack_trace" : "ElasticsearchException[java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [leadDeveloper.geoIp.city_name] doesn't exist]; nested: IllegalArgumentException[java.lang.IllegalArgumentException: field [leadDeveloper.geoIp.city_name] doesn't exist]; nested: IllegalArgumentException[field [leadDeveloper.geoIp.city_name] doesn't exist];\r\n\tat org.elasticsearch.ingest.CompoundProcessor.newCompoundProcessorException(CompoundProcessor.java:188)\r\n\tat org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:134)\r\n\tat org.elasticsearch.ingest.Pipeline.execute(Pipeline.java:97)\r\n\tat org.elasticsearch.ingest.IngestService.innerExecute(IngestService.java:479)\r\n\tat org.elasticsearch.ingest.IngestService.access$100(IngestService.java:68)\r\n\tat org.elasticsearch.ingest.IngestService$4.doRun(IngestService.java:408)\r\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723)\r\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\r\n\tat java.base/java.lang.Thread.run(Thread.java:834)\r\nCaused by: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [leadDeveloper.geoIp.city_name] doesn't exist\r\n\t... 11 more\r\nCaused by: java.lang.IllegalArgumentException: field [leadDeveloper.geoIp.city_name] doesn't exist\r\n\tat org.elasticsearch.ingest.common.RenameProcessor.execute(RenameProcessor.java:68)\r\n\tat org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:124)\r\n\t... 9 more\r\n"
      }
    ],
    "type" : "exception",
    "reason" : "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [leadDeveloper.geoIp.city_name] doesn't exist",
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "java.lang.IllegalArgumentException: field [leadDeveloper.geoIp.city_name] doesn't exist",
      "caused_by" : {
        "type" : "illegal_argument_exception",
        "reason" : "field [leadDeveloper.geoIp.city_name] doesn't exist",
        "stack_trace" : "java.lang.IllegalArgumentException: field [leadDeveloper.geoIp.city_name] doesn't exist\r\n\tat org.elasticsearch.ingest.common.RenameProcessor.execute(RenameProcessor.java:68)\r\n\tat org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:124)\r\n\tat org.elasticsearch.ingest.Pipeline.execute(Pipeline.java:97)\r\n\tat org.elasticsearch.ingest.IngestService.innerExecute(IngestService.java:479)\r\n\tat org.elasticsearch.ingest.IngestService.access$100(IngestService.java:68)\r\n\tat org.elasticsearch.ingest.IngestService$4.doRun(IngestService.java:408)\r\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723)\r\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\r\n\tat java.base/java.lang.Thread.run(Thread.java:834)\r\n"
      },
      "stack_trace" : "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [leadDeveloper.geoIp.city_name] doesn't exist\r\n\tat org.elasticsearch.ingest.CompoundProcessor.newCompoundProcessorException(CompoundProcessor.java:188)\r\n\tat org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:134)\r\n\tat org.elasticsearch.ingest.Pipeline.execute(Pipeline.java:97)\r\n\tat org.elasticsearch.ingest.IngestService.innerExecute(IngestService.java:479)\r\n\tat org.elasticsearch.ingest.IngestService.access$100(IngestService.java:68)\r\n\tat org.elasticsearch.ingest.IngestService$4.doRun(IngestService.java:408)\r\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723)\r\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\r\n\tat java.base/java.lang.Thread.run(Thread.java:834)\r\nCaused by: java.lang.IllegalArgumentException: field [leadDeveloper.geoIp.city_name] doesn't exist\r\n\tat org.elasticsearch.ingest.common.RenameProcessor.execute(RenameProcessor.java:68)\r\n\tat org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:124)\r\n\t... 9 more\r\n"
    },
    "header" : {
      "processor_type" : "rename"
    },
    "stack_trace" : "ElasticsearchException[java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [leadDeveloper.geoIp.city_name] doesn't exist]; nested: IllegalArgumentException[java.lang.IllegalArgumentException: field [leadDeveloper.geoIp.city_name] doesn't exist]; nested: IllegalArgumentException[field [leadDeveloper.geoIp.city_name] doesn't exist];\r\n\tat org.elasticsearch.ingest.CompoundProcessor.newCompoundProcessorException(CompoundProcessor.java:188)\r\n\tat org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:134)\r\n\tat org.elasticsearch.ingest.Pipeline.execute(Pipeline.java:97)\r\n\tat org.elasticsearch.ingest.IngestService.innerExecute(IngestService.java:479)\r\n\tat org.elasticsearch.ingest.IngestService.access$100(IngestService.java:68)\r\n\tat org.elasticsearch.ingest.IngestService$4.doRun(IngestService.java:408)\r\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723)\r\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\r\n\tat java.base/java.lang.Thread.run(Thread.java:834)\r\nCaused by: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [leadDeveloper.geoIp.city_name] doesn't exist\r\n\t... 11 more\r\nCaused by: java.lang.IllegalArgumentException: field [leadDeveloper.geoIp.city_name] doesn't exist\r\n\tat org.elasticsearch.ingest.common.RenameProcessor.execute(RenameProcessor.java:68)\r\n\tat org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:124)\r\n\t... 9 more\r\n"
  },
  "status" : 500
}

given the following index operation:

PUT http://127.0.0.1:9200/project/doc/nest-fluent-ef0c8aa6?pretty=true&error_trace=true&routing=nest-fluent-ef0c8aa6&refresh=true&pipeline=pipeline-8da9cd6e HTTP/1.1
Accept: application/json
Content-Type: application/json
Content-Length: 451
Host: 127.0.0.1:9200

{
  "curatedTags": [
    {
      "added": "2015-06-06T12:01:02.123",
      "name": "x"
    }
  ],
  "join": "project",
  "lastActivity": "2015-06-06T12:01:02.123",
  "leadDeveloper": {
    "ipAddress": "193.4.250.122",
    "id": 1
  },
  "name": "nest-fluent-ef0c8aa6",
  "numberOfContributors": 0,
  "startedOn": "2015-06-06T12:01:02.123",
  "state": "Stable",
  "type": "project",
  "visibility": "Public"
}

[elasticmachine]

Pinging @elastic/es-core-features

[martijnvg]

I suspect that the due to the database update done via #33840, the ip address 193.4.250.122 returns different data. These changes happen when the geoip database is updated and I don't think we can do much about it. (when the database update change is made, some of the geoip processor tests almost always need to be modified).

[Mpdreamz]

One possible fix could be that we always inject the properties whether or not they have a value or not. This would at least not require the user the update their pipeline(s) in a minor.

That fix by itself would be breaking but would be a nice behaviour from 7.0 onwards.

[martijnvg]

@Mpdreamz I agree, that makes sense.

[martijnvg]

@Mpdreamz This was discussed during the @elastic/es-core-features weekly meeting and we think the geoip processor should not inject empty/null properties in case the geoip database has no entry for an ip address (or some fields of an entry are missing). Instead a pipeline should be resilient against missing fields and this can be done today by configuring ignore_missing flag on the rename processor.

[Mpdreamz]

Ok fair enough, the.docs do state they are dependent on whether they have a value or not..