Extra Lucene DocValueExistsQuery fired - due to nested mapping and primary_terms

[jeffreynscrbdee]

Elasticsearch version (bin/elasticsearch --version): 6.3.2

Description of the problem including expected versus actual behavior:

After upgrading from 6.0.0 to anything >= 6.1.0, we are getting an extra DocValueFieldsExistsQuery [field=_primary_term] being called. This is an extra underlying query that impacts performance. This seems to be potentially introduced at #27469 which resolves #24362 based on the release notes.

Steps to reproduce:

1. Mapping is defined as

{
    "mappings": {
        "item": {
            "item": {
                "_all": {
                    "enabled": false
                },
                "properties": {
                    "id": {
                        "type": "integer"
                    },
                    "document_type": {
                        "type": "keyword",
                        "doc_values": false
                    },
                    "language": {
                        "type": "keyword",
                        "doc_values": false
                    },
                    "authorships": {
                        "type": "nested",
                        "properties": {
                            "user_id": {
                                "type": "keyword",
                                "doc_values": false
                            },
                            "contribution_types": {
                                "type": "keyword",
                                "doc_values": false
                            }
                        }
                    }
                }
            }
        }
    }
}

2. Data inserted is defined as

{
    "document_type": "5",
    "language": "1"
}

3. The query which, when profiled, spits out the extra DocValueFieldsExistsQuery (_primary_term) is

{
    "profile": true,
    "query": {
        "bool": {
            "filter": [{
                "terms": {
                    "document_type": [1, 5]
                }
            }, {
                "terms": {
                    "language": [1]
                }
            }]
        }
    }
}

The partial results from profiling the query above are

(#ConstantScore(document_type:1 document_type:5) #ConstantScore(language:1)) #DocValuesFieldExistsQuery [field=_primary_term]"

• the DocValuesFieldExistsQuery is causing extra overhead which adds to the response time.

Some extra findings - the extra DocValuesFieldExistsQuery only surfaces when there is a nested mapping and all the filter arguments are "terms" which means all array-type. For example - if I change language to be a term-query which filters on just 1, the extra Lucene query does not fire.

So if there is not a nested mapping, this extra query never fires. If there exists a nested mapping, we can get around it by including a singular term query.

I also profiled 6.0.0 and 6.0.1. 6.0.1 introduces another query which seems like a prerequisite for this new change. Profiling the query above in ES 6.0.1 produces this partial profile output

(#ConstantScore(document_type:1 document_type:5) #ConstantScore(language:1)) #(#*:* -_type:__*)

The #*:* -_type:__* adds unnecessary overhead as well - it seems to be fire a MatchAllDocsQuery. The current solution seems to be going back to 6.0.0.

The performance impacts for us are the following:

1. For our smaller recommendations index, we are getting avg response times of 100+ ms vs 50 ms without the extra query.
2. For one of our bigger indices (300 million docs), our avg response times are 700 ms vs 350 ms.

The query example above is simplified compared to our production use case, but the behavior still remains consistent.

Is this behavior intentional?

[elasticmachine]

Pinging @elastic/es-search-aggs

[jimczi]

Is this behavior intentional?

Yes and it always existed. What changed in 6x is that we don't index the _type field anymore for index created in this version. The _type field was used for two purposes:

• Separate root documents of different types in a single index when multiple types were allowed per mapping.
• Index the path of nested document.

When we run a query against a specific index we need to ensure that only root documents are returned. To achieve this we used to rely on the _type field, if multiple types are present in the index and the query is against a specific type then we are sure that only root document can match. If the query targets all _type then we need to filter nested document from the response, this was done with the #*:* -_type:__* query which removes any document with a _type that starts with __ (this is the prefix we use for nested document path). However this query is quite expensive since it can only work as a negative clause (all documents but...) so in 6.1.0 we made a small change to allow fast filtering of nested documents. This is #27469 which relies on the fact that we don't need to index a metadata field in the nested documents and that this absence can be used to build an efficient filter for removing nested documents (the DocValuesExists query).

The performance impacts for us are the following:
For our smaller recommendations index, we are getting avg response times of 100+ ms vs 50 ms without the extra query.
For one of our bigger indices (300 million docs), our avg response times are 700 ms vs 350 ms.

Which version did you test ? 6.1.0 with the DocValuesExists query or the old version with the negative clause ? I suspect that it's the latter which is expensive compared to a simple DocValuesExists query. I would expect a small overhead when the DocValuesExists query is used so can you please clarify what you tested.

Some extra findings - the extra DocValuesFieldExistsQuery only surfaces when there is a nested mapping and all the filter arguments are "terms" which means all array-type. For example - if I change language to be a term-query which filters on just 1, the extra Lucene query does not fire.

In 6x we have a new optimization that checks if the query can match nested documents and if not we don't add the nested filter query. The check looks for mandatory term or range query that target a root document field. If we find one we can safely assume that no nested documents can match the main query. We can add support for terms query as well but as I said above if you reindex your data in 6.1.0 you should be able to use the fast version (DocValuesExistsQuery) and the overhead should be small.

[jeffreynscrbdee]

Hey @jimczi , thanks for the quick response!

The difference in response times I mentioned above were specifically for version 6.3.2. To further clarify, we were getting faster times by changing the terms to term which used the new optimization you mentioned in your last paragraph. The problem I see is that this might not always be a feasible solution. For example, we might have filters that are all array-based which means we will always fire this additional query.

So there is a considerable difference for us. We are noticing heavy overhead with the additional DocValuesExistsQuery. I only mentioned #*:* -_type:__* as an extra finding because I was comparing 6.0.0, 6.0.1, and 6.1.0 to see when this new Lucene query was introduced.

It would be really great if we can add the optimization to check for terms to ignore the nested filter query. This would improve our performance considerably.

[jpountz]

+1 to make it work with terms as well.