mapper_parsing_exception on `dynamic: runtime` between object and non-object

[PhaedrusTheGreek]

ES 7.11.1

When using dynamic:runtime, I am able to continue "indexing" even when having encountered a runtime mapping conflict:

PUT test-index-1
{
  "mappings": {
    "dynamic": "runtime"
  }
}

PUT test-index-1/_doc/1
{
  "field1": 22
}

PUT test-index-1/_doc/2
{
  "field1": "string"
}

Also, I can search the data if I know what type i'm looking for:

GET test-index-1/_search
{
  "runtime_mappings": {
     "my_field": {
       "type": "keyword",
       "script": {
         "source": "emit(params._source.field1.toString())"
       }
     }
   },
   "query": {
     "term": {
       "my_field": {
         "value": "string"
       }
     }
   }
   
}

However when Elasticsearch encounters a runtime field conflict between object and non-object mappings, a rejection occurs:

PUT test-runtime
{
  "mappings": {
    "dynamic" : "runtime"
  }
}

PUT test-runtime/_doc/1
{
  "my-obj": {
    "val": 1
  }
}

PUT test-runtime/_doc/1
{
  "my-obj": "not-object"
}

{
  "error" : {
    "root_cause" : [
      {
        "type" : "mapper_parsing_exception",
        "reason" : "object mapping for [my-obj] tried to parse field [my-obj] as object, but found a concrete value"
      }
    ],
    "type" : "mapper_parsing_exception",
    "reason" : "object mapping for [my-obj] tried to parse field [my-obj] as object, but found a concrete value"
  },
  "status" : 400
}

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[javanna]

Thanks for opening this one @PhaedrusTheGreek . I am wondering what the ideal user experience would be here.

Runtime fields don't affect document parsing, as they are loaded at runtime. In the first scenario you mentioned, the first document would drive how the runtime field is mapped (meaning what type it gets) but following documents with a different type for the same field do get accepted. What happens when a field is mapped as a number but its values are not a number and we perform search on it? At the moment malformed values are ignored, effectively the same behaviour as when ignore_malformed is set to true for indexed fields. When we previously discussed this we said we don't want to fail at search time, and it is not necessary to fail at index time, because effectively we are not indexing anything, hence ignoring malformed values was what we went for. It is certainly good to re-evaluate this and see whether there are scenarios where we'd like to enforce consistency on types also for runtime fields. In general, runtime fields allow for flexibility: for instance if you realize later that a field is not always a number in your documents, you could downgrade it to be a keyword field so that all values are extracted as strings, but then you lose the ability to query them as numbers, in which case strings need to be ignored.

The discrepancy around objects is because even if an object holds fields that are all runtime, the object itself will be mapped as an object under the properties section. That means that a subsequent document that has the same field as a leaf field, will cause an error. I can see how this is confusing, and deserves more discussion with the team.

[PhaedrusTheGreek]

dynamic: false can promise "at least source storage". dynamic: runtime has a great advantage over dynamic: false, in that fields are discoverable in Kibana, which makes all the difference when ingesting and discovering complex and undocumented structures in logging environments. The potential (and probably the expectation) of dynamic: runtime is to provide both, which gives users an "always stored source, almost always discoverable fields" experience.

I wonder if instead of failing, we could flag a mapping as having conflict so that this could be indicated to the user along with resolution workflow.

[javanna]

Am I reading it correctly that you find the behaviour around malformed values expected, but the error around object becoming another type unexpected? In the case of flagging, would you then flag both scenarios?

[PhaedrusTheGreek]

I would consider an object to be a malformed primitive value and vice versa, and expect the handling to be the same as between malformed values. In either case, non-blocking and flagged.