Versioning aware scripting

[bpintea]

QL makes use of scripts referencing internal functions; these scripts and functions can be occasionally updated for things like improving accuracy or fix a defect.

In the context of a mixed-versions cluster (such as during an upgrade) and such a function/script update, it can happen that part of the result will be returned with one version of the function/script (the old one), while other part with a different version (the new one).
Ideally, we would have a mechanism in place that could help mitigate this situation. One possible example would be the use of a serial number or a script/function version that would allow either rerouting of requests or fail safe.

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[rdnm]

@jdconrad @stu-elastic @rjernst - any thoughts on how we could achieve this now we've had some time to think about it?

[stu-elastic]

Some of our thoughts from #19162 (comment) may be relevant.

I know that's for stored scripts, but the thinking in 3&4 may be relevant.

[stu-elastic]

There's nothing we're going to do in scripting for now.

Some options for mitigation:

• Translate the QL script to painless per shard
• Use the transport layer bwc techniques to inform the translation

I'm going to close this, feel free to reopen it if you'd like to discuss it further.

[bpintea]

The above proposals might be problematic, since the node handling the QL request, after transpiling the *QL to DSL/Painless, uses the node-local client to perform the search. This client then distributes the request to whatever shard needs to be hit. So QL remains at a too higher level to either be aware of the nodes in the cluster (i.e. shard versions) that'll be eventually queried or have access to the transport layer.

This search client returns the assembled results to QL, so any version information from each node is stripped by the time QL gets the "full" answer. The only way to get the remote node version in QL is if that node fails with a specific error containing the version information (example).

Side note: Painless is used by QL not just for projections (i.e. modify the end-results before returning them to the end-client), but also filtering or in aggregations.

Wanted to log some other options discussed on the topic:

• embed the version of the node doing the transpiling as Painless argument(s) provided to QL's functions. A node running an older version could then decide to fail the request based on this. A node running a newer version could either know how to answer to an older node, or also fail the request.
• build a functions manifest, per version; multiple manifests - all previous versions (starting at some point) - would then be available statically to each node. An older code would simply translate according to its latest manifest. A newer node could generate the Painless code based on the manifests specific to the old version in the request.

But I believe we also wanted to asses how the Painless-in-a-JAR would impact this bwc need in QL and how that would look longer term, @rdnm? Please feel free to reclose if that is no longer the case.

[rjernst]

Let me expand a bit on the suggested alternative approach.

First, let me restate the problem. If my understanding is correct, QL currently translates (or transpiles as you put it) the *QL request into a normal Elasticsearch query. It uses the NodeClient to run that query, which will then act as a coordinating node, sending out shard requests for the query to the appropriate nodes. When Elasticsearch is being upgraded, some of these nodes may be an older version. Yet the QL code has already transpiled parts of its query into Painless. So if anything has changed inside Painless across these versions, errors can occur on some of the shards.

Notice that the rest of the query, sent via the NodeClient, does not have this problem. Methods are renamed, APIs changed, etc across versions within the codebase. The queries, and all of the inter node communication within Elasticsearch, version the communication protocol, so that if a newer node is talking to an older node, the data is adjusted to the older expected format.

The suggestion we made that Stu noted above was actually two parts, not two options. In short, change QL to have its own queries. Those queries will then be converted into local queries on each shard, at which time the Painless portion can be generated. The transport bwc layer is how this would be achieved. And as a side note, this very change in architecture would be needed in the future anyways to enable QL to utilize Painless as a library, in order to add their own optimizations like method inlining.

Regarding the other options we had discussed a bit:

embed the version of the node doing the transpiling as Painless argument(s)

While this could be possible, it requires exposing the node version within Painless, and with that would want to come all of the utility methods for comparing, parsing, etc. It's not something we want to do. Additionally, it seems similar in theory as the suggested solution, at least as far as both would contain versioned logic for which code to execute, just a difference in where that versioning logic lives.

build a functions manifest, per version; multiple manifests

This would be a very large amount of work, and semantics would be unclear (what about plugins?). It's not a general purpose scripting solution.

The longer term solution is moving QL to have it's own query type(s), which can then utilize Painless internally, whether that is via string generation with the scripting framework as it is right now, or using Painless-in-a-jar as a backend to generate Java code directly.

[bpintea]

If my understanding is correct [...]

Indeed, the problem is as you've summarised it.

Notice that the rest of the query, sent via the NodeClient, does not have this problem. [...] if a newer node is talking to an older node, the data is adjusted to the older expected format.

QL doesn't have any logic for handling different nodes, version-dependent either, that's why an ideal solution to this issue for QL, as a Painless user, emerged as an equally transparent behaviour as the versioned protocol / bwc layer for search.

In short, change QL to have its own queries.

We've discussed it internally, didn't come to a detailed view on how this would work, but it might be admittedly a solution. However, it would probably come either with its own bwc issues (old nodes would still not support a new query type) or at least a complex bwc layer, very specialised to QL-Painless interaction. We couldn't decide on this solution at this point as it'd require core changes in QL, we'd extend efforts into a direction which isn't QL's area of focus a.t.m. and ...

this very change in architecture would be needed in the future anyways to enable QL to utilize Painless as a library

... this bit is still an unknown.

In any case, many thanks for the extended explanations and offered alternatives, @rjernst!