Collapse field types into "families" of field types in `_field_caps`.

[jpountz]

Both Kibana and SQL have logic on top of the _field_caps API to detect when a conflict is harmful versus when it's not. Part of how it works is by grouping field types into families of fields: conflicts are fine as long as the types across indices are all of the same family.

Should we move this logic to _field_caps instead?

Here is an example of how we could collapse fields:

• number: long, integer, double, float, scaled_float, ...
• date: date, date_nanos
• keyword: keyword, constant_keyword, wildcard
• text: text
• geo: geo_point, geo_shape

Most other fields would be their own family as they don't share properties with other fields, e.g. boolean or flattened.

So for instance, the below response today is considered a conflict:

{
    "indices": ["index1", "index2", "index3", "index4", "index5"],
    "fields": {
        "rating": { 
            "double": {
                "searchable": true,
                "aggregatable": true,
                "indices": ["index1", "index2"]
            },
            "float": {
                "searchable": true,
                "aggregatable": true,
                "indices": ["index3", "index4"],
            }
        }
    }
}

While it wouldn't be one if we collapsed double and float into a number "family".

[elasticmachine]

Pinging @elastic/es-search (:Search/Mapping)

[costin]

Having some type of "category" would be useful however I'm not sure whether in the end it would be useful across all consumers.
For example for SQL there's a big difference in reporting a long vs a double as they have different size, precision ,etc...
Currently in case of a merge we do consider the field conflicting since it would be a merge between similar but yet different types. This is something that can be changed as we chose a different merging strategy however it's valuable to have the consumer decide on that instead of pushing that into field_caps.

For example, if a field is mapped as a byte, short, integer, long and double in case of merging what would be winning type and why?

[jpountz]

This is something that can be changed as we chose a different merging strategy however it's valuable to have the consumer decide on that instead of pushing that into field_caps.

I have mixed feelings on this. We're doing everything so that field types are as transparent as possible, e.g. long fields accept range queries that have floating-point bounds and do the right thing. Yet introducing new numeric fields requires changes in our SQL layer requires changes, which feels wrong to me.

For example, if a field is mapped as a byte, short, integer, long and double in case of merging what would be winning type and why?

How do you do it in SQL today?

[astefan]

In SQL, we depend on field_caps almost entirely and while keyword and constant_keyword can be regarded as "equivalent" when merging mappings, there are scenarios that need to know what a field type truly is. Think about DESCRIBE index_alias and DESCRIBE single_index. The first one will consider the equivalence between keyword and constant_keyword and the field with the same name will be displayed as keyword, while the second command will show constant_keyword field type.

Note: keyword and constant_keyword is the only pair of field types that are considered equal in the merging mappings scenario.

With numeric fields, apart from the scenario above where we do not make them "equal" in any case, knowing which type is which is important when we take the _source and re-parse that numeric field to be displayed back to the user. A small note here: a scaled_float is not extracted from _source.

[mattkime]

When you say that kibana performs this kind of merge, I think you're talking about kibana index patterns. I'm curious if there are other parts of kibana in which the specific field types are important.

Here's the related kibana code - https://github.com/elastic/kibana/blob/master/src/plugins/data/common/kbn_field_types/kbn_field_types_factory.ts

Would this change merge collapsible fields in such as way that we would no longer know the types comprising them?

[jpountz]

@mattkime If knowing the actual types is a requirement for you, it would be doable. Are there places where you need to know about the actual types today?

[ppisljar]

i don't think we are interested in the actual elastic search type in kibana. am i missing something @timroes ?

[timroes]

We're storing the actual ES types and so removing that would also be a breaking change. In terms of what we're using them for: I don't have a complete list, but I think it was mainly introduced for KQL, since we needed to make some decision between text and keyword fields for the way we build the query, but I can't find that specific code right now. Another example seems to be field formatters where we are using it.

Even if we could solve those without, I think we'll always have the need. One example that would come to my mind is e.g. support for significant text aggregation (elastic/kibana#31614). We can only achieve that if we have the real ES types available, since our grouped "String" type is not enough and we also explicitly need to bypass text fields not being aggregatable by default for allowing them for this aggregation.

I don't think removal of esTypes is something we want to do at the moment and we'll run in the future into more problems and features we cannot build with it (again). Also if we would decide to remove it, I think this should be some kind of breaking change.