Handling new APIs in a mixed versions cluster

[astefan]

SQL/EQL is an internal user of Elasticsearch and a query request is translated in a _search request on the receiving node, then sent via transport to the other nodes in the cluster. Depending on the version of the node receiving the request, the way the query dsl will look like can differ. With the introduction of the "fields" API, the query dsl will stop using docvalue_fields and _source filtering in favor of "fields" and simplify the overall response handling logic.

In a mixed versions cluster where several nodes are on the old version (where "fields" API doesn't exist) and others are on the new one the search request on those nodes will look differently, as well as the response coming back from ES. In this scenario a client (internal - SQL -, or external) will potentially have to deal with an array of response formats, which adds tremendous complexity to code base in *QL's case and be impossible to adopt the new API.

Some open questions to consider:

• should this behavior be acceptable and users aware of these edge cases in a rolling upgrade scenario?
• should this mixed versions support be a must and ES do a best effort to adapt various response formats to a common one?

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[costin]

Related to this is Point-In-Time and its behavior/guarantees in mixed clusters.

[jtibshirani]

We decided to move this to a team-wide discussion, as has implications beyond the search area.

[astefan]

We discussed this in a team-wide meeting and fleshed out few details in a second smaller meeting. Below it's a summary. CC @costin @jimczi

The _search API will be able to detect if a search request will run, based on minimum version compatibility, on nodes having the same version. If not, it will fail fast and, based on the failure that will contain a version hint, the SQL plugin will react by sending/redirecting the SQL query (at "transport" protocol level) to another node that will be able to create the search query DSL supported by the nodes having the same version. The _search API will reply with the ES version that the query should run on and SQL will pick a new ES node to redirect the SQL request to, based on this version.

Alternatively or, better said, after the SQL code that handles the redirection based on the optimal ES version gets implemented, we could formalize/generalize the redirection code that can get implemented ES wide and be used by other internal plugins.

Open question: could SQL detect if a query will need redirection before actually running the _search request?
Possible answer: it can not since the list of shards a _search request will run on cannot be decided without the _search query itself.

Note: this change shouldn't be confused with HTTP versioning or Transport versioning. This is a mechanism for supporting
new search API features introduced in point releases in a safe manner during upgrades.