Introduce flag for testing CCS compatibility #81809
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
CCS works with a subset of APIs and features depending on the versions of the clusters being communicated with. Currently we limit this CCS compliance to one minor version backward and one minor forward. This change adds a flag that can be turned on to test if a request sent to one of the endpoints that are supporting CCS is serializable to a cluster that is on one minor version backward. We do this by trying to serialize the request to a steam with that earlier version. Features and components that are not supported in that version should throw errors upon atempted serialization to indicate they are not compatible with that version. In addition we need components extending NamedWriteable (e.g. new queries) to also error when they are written to a stream that has a version before the version they are released in.
|
Caveat: I didn't add checks to the QL rest endpoints and the Vector tile search search yet, althought they are mentioned in the supported APIs because I want to discuss with the respective teams first. |
javanna
requested changes
Jan 14, 2022
.../lang-mustache/src/main/java/org/elasticsearch/script/mustache/RestSearchTemplateAction.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/elasticsearch/common/io/stream/NamedWriteable.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/elasticsearch/common/io/stream/VersionCheckingStreamOutput.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/elasticsearch/rest/action/search/CCSVersionCheckHelper.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/elasticsearch/rest/action/search/CCSVersionCheckHelper.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/elasticsearch/rest/action/search/CCSVersionCheckHelper.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/elasticsearch/common/io/stream/VersionCheckingStreamOutput.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/elasticsearch/rest/action/search/CCSVersionCheckHelper.java
Outdated
Show resolved
Hide resolved
server/src/test/java/org/elasticsearch/rest/action/search/RestSearchActionTests.java
Outdated
Show resolved
Hide resolved
server/src/test/java/org/elasticsearch/rest/action/search/RestSearchActionTests.java
Outdated
Show resolved
Hide resolved
javanna
reviewed
Jan 18, 2022
server/src/main/java/org/elasticsearch/common/io/stream/VersionCheckingStreamOutput.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/elasticsearch/rest/action/search/CCSVersionCheckHelper.java
Outdated
Show resolved
Hide resolved
server/src/test/java/org/elasticsearch/rest/action/search/RestSearchActionTests.java
Outdated
Show resolved
Hide resolved
cbuescher
changed the title
cbuescher
added
:Search/Search
|
Pinging @elastic/es-search (Team:Search) |
added 8 commits
January 18, 2022 13:56
|
Heads up to @elastic/kibana-operations and @stacey-gammon that this PR is merged now and starting with 8.1 you should be able to set the 'search.check_ccs_compatibility' setting in the config of a test cluster to enable the CCS compatibility checks on the coordinating node. Since we had no new features added since 8.0 we don't espect any errors when the setting is activated, please let us know if you have questions or once you start using this for testing. |
cbuescher
pushed a commit
to cbuescher/elasticsearch
that referenced
this pull request
Feb 1, 2022
In elastic#81809 we introduced a mechanism to check searializability of search request to earlier version nodes already on the coordinating node. This requires knowledge about the version that NamedWritable classes have been introduced, which is why we started moving classes that are used inside the search request under the VersionedNamedWriteable interface to make sure future additions implement the mthod that provides the version information. This change moves ScoreFunctionBuilders which are used in function score queries under that interface. Since all these builders have been around before 7.0 we return the empty version constant for them.
This was referenced Feb 1, 2022
cbuescher
pushed a commit
that referenced
this pull request
Feb 1, 2022
In #81809 we introduced a mechanism to check searializability of search request to earlier version nodes already on the coordinating node. This requires knowledge about the version that NamedWritable classes have been introduced, which is why we started moving classes that are used inside the search request under the VersionedNamedWriteable interface to make sure future additions implement the mthod that provides the version information. This change moves ScoreFunctionBuilders which are used in function score queries under that interface. Since all these builders have been around before 7.0 we return the empty version constant for them.
cbuescher
pushed a commit
that referenced
this pull request
Feb 3, 2022
Supporting #81809, we changed query builders to implement 'VersionedNamedWriteable' to be able to detect when new query builders under the search enpoint are introduced and also to force new implementations to overwrite 'getMinimalSupportedVersion' with a current release version. This change removes the default implementation in the QueryBuilder interface and replaces it with individual implementations in the currently existing query builders. For builders that have been around for longer than 7.0 (the earliest verison constant we currently have around) we use Version.V_EMPTY which sorts always before any other declared version.
cbuescher
pushed a commit
to cbuescher/elasticsearch
that referenced
this pull request
Feb 7, 2022
In elastic#81809 we introduced a mechanism to check searializability of search request to earlier version nodes already on the coordinating node. This requires knowledge about the version that NamedWritable classes have been introduced, which is why we started moving classes that are used inside the search request under the VersionedNamedWriteable interface to make sure future additions implement the mthod that provides the version information. This change moves aggregation builders under that interface. Most builders have been around long before version 7.0, so we return an empty version tag for them. The newer ones return the version they were first released in.
cbuescher
pushed a commit
that referenced
this pull request
Feb 8, 2022
In #81809 we introduced a mechanism to check searializability of search request to earlier version nodes already on the coordinating node. This requires knowledge about the version that NamedWritable classes have been introduced, which is why we started moving classes that are used inside the search request under the VersionedNamedWriteable interface to make sure future additions implement the mthod that provides the version information. This change moves aggregation builders under that interface. Most builders have been around long before version 7.0, so we return an empty version tag for them. The newer ones return the version they were first released in.
cbuescher
pushed a commit
that referenced
this pull request
Feb 9, 2022
In #81809 we introduced a mechanism to check searializability of search request to earlier version nodes already on the coordinating node. This requires knowledge about the version that NamedWritable classes have been introduced, which is why we started moving classes that are used inside the search request under the VersionedNamedWriteable interface to make sure future additions implement the mthod that provides the version information. This change moves pipeline aggregation builders under that interface. Most builders have been around long before version 7.0, so we return an empty version tag for them. The newer ones return the version they were first released in.
cbuescher
pushed a commit
to cbuescher/elasticsearch
that referenced
this pull request
Feb 16, 2022
In elastic#81809 we introduced a mechanism to check searializability of search request to earlier version nodes already on the coordinating node. This requires knowledge about the version that NamedWritable classes have been introduced, which is why we started moving classes that are used inside the search request under the VersionedNamedWriteable interface to make sure future additions implement the mthod that provides the version information. This change moves the InferenceConfigUpdate and implementing classes under that sub-interface. I have used the versions they were first released in looking at the pull requests that introduced the classes.
cbuescher
pushed a commit
that referenced
this pull request
Feb 16, 2022
In #81809 we introduced a mechanism to check searializability of search request to earlier version nodes already on the coordinating node. This requires knowledge about the version that NamedWritable classes have been introduced, which is why we started moving classes that are used inside the search request under the VersionedNamedWriteable interface to make sure future additions implement the mthod that provides the version information. This change moves the InferenceConfigUpdate and implementing classes under that sub-interface. I have used the versions they were first released in looking at the pull requests that introduced the classes.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
CCS works with a subset of APIs and features depending on the versions of the clusters being communicated with.
Currently we limit this CCS compliance to one minor version backward and one minor forward.
This change adds a setting useful for testing in clients like Kibana that can be turned on to check if a search request
sent to one of the endpoints that are supporting CCS is compatible with a cluster that is on one minor version back.
We do this by trying to serialize the request to a stream with the earlier version. Features and components that are
not supported in that version should throw errors upon atempted serialization to indicate they are not compatible.
In addition we need components extending NamedWriteable (e.g. new queries) to also error when they are written to a
stream that has a version before the version they were released.