Description
Cross-cluster search allows to query a remote cluster that can be on a different version than the local cluster.
The cluster responsible to send the remote request can be on 7.12 for instance and the remote cluster on 6.8.
The logic to send a query to such configuration should be to use features and options that are available on 6.8.
However it is complex for clients, even internally (Kibana), to detect these cases consistently.
The first issue is that we don't have a consistent way to handle a feature in the search API that is not available in a remote server. When we detect this in the transport layer, most of the time we act as if the option was not there. That's a case of silent breakage.
In such case I can see two options:
- Throw a clear error at the shard level that the feature cannot be used on an older node.
- Skip the option in the older node but add a warning in the response that we skipped it.
The skip part could be useful in a best effort scenario. For instance the case_insensitive option of the terms queries could be skipped in older nodes, we'd return matches without applying the option and the warning in the response would indicate which shard/index is concerned.
The error path would be reserved for cases where there is no other alternative. The error message can be clear and should mention the feature that the old node cannot handle.
I opened the issue to discuss how we can consistently add new options and features compatible with cross cluster search.
Having a clear warning/error for the mixed-version case would help to clarify why something is broken. That also shifts the responsibility to Elasticsearch to provide a comprehensive message on this complex configuration.