Description
Systems/clients that wish to interact with Elasticsearch are required to do so over the REST API. Breaking changes to the REST API (in major versions) can introduce challenges for these systems/clients.
The preferred upgrade path is 1) upgrade Elasticsearch 2) upgrade the systems/clients that are interacting with Elasticsearch. However, major version upgrades are more likely 1) upgrade systems/clients to be compatible with the next major version of Elasticsearch 2) upgrade Elasticsearch. Upgrading the systems/clients that interact with Elasticsearch before upgrading Elasticsearch can create an high orchestration upgrade event. The administrators of Elasticsearch need to know that all systems that interact with Elasticsearch are compatible with the new major version before they can upgrade.
Rest API version compatibility seeks to address better de-coupling of systems/clients and server upgrades. Rest API version compatibility will ensure that requests and responses that worked in the prior major version will continue to function in the next major version without error (but may behave differently). The intent is to allow Administrators to upgrade the Elasticsearch server to new major versions without requiring (or at least minimal) changes to the systems/clients that talk to Elasticsearch. Further, Rest API version compatibility will not require specific deployment architectures or complex configuration. Rest API version compatibility will be enabled/disabled per HTTP request via an HTTP header.
Rest API version compatibility can be defined along two axis: 1) shape 2) behavior.
Shape
Shape is being defined as the structure of the HTTP request. Shape includes the JSON body for the request/response, the HTTP verb (GET, POST, etc.), the HTTP path (/foo/bar), HTTP parameters (?foo=bar), etc. Changes to the shape of request and responses (with minor exception) will be covered by REST API version compatibility.
For example, in 8.0 REST paths that started _xpack will no longer be available and will be replaced with a similar path but without the _xpack in the URL. If a system/client relied upon any of those URLs with _xpack in the path, those systems/clients would need to be upgraded prior to upgrading Elasticsearch from 7.x to 8.x.
With Rest API version compatibility (assuming it was requested via the header), those paths will still be honored since they were valid in 7.x. Any shape honored in 7.x will be honored in 8.x when Rest API version compatibility is requested. The request header that enables Rest API version compatibility can be requested to be compatible with a specific major version, for example, a request can request compatibility with v7. When Elasticsearch server is running 7.x and compatibility is requested for v7, the header will essentially be ignored. However, as soon as Elasticsearch is upgraded (from 7.x to 8.x) and the system/client is still sending a header requesting compatibility with v7, the 8.x server will read that header and ensure that any shape elements that worked in 7.x will continue to work in 8.x. This type of compatibility is a close cousin to traditional wire or binary compatibility.
For these shape based breaking changes across major versions, Rest API version compatibility allows the server to be upgraded independently to any required changes for the client.
Behavior
Rest API version compatibility as described here is distinctly different then most SAAS version-ed APIs. SAAS offerings can allow for stronger guarantees for how a service behind the version of REST API behaves then software you can download and run yourself. With Rest API version compatibility, the shape of the REST API (with minor exception) will be guaranteed to be compatible with the previous minor version, however, the behavior of the server may differ from the last prior major version.
For example, the default number of shards per index was changed in major version. Had Rest API version compatibility existed then it would have no impact on that change since that change was strictly about how Elasticsearch behaves. Many changes across majors (and minors) are changes to the behavior of the service and will be not influenced by REST API version compatibility.
Another example is that in 8.0 many of the deprecated discovery.zen are removed since a new cluster coordination subsystem was implemented in 7.x and the old one will be completely removed in 8.0. Since cluster coordination (and the related settings) is a behavior, not an REST API, it is not part Rest API version compatibility.
Shape && Behavior
Quite often shape and behavior change go hand-in-hand. Not all changes to the shape are as simple as a changing URL. Many of the breaking changes to the REST API also include some form of behavior change as well.
For 8.0 the most notable change that encompasses both shape and behavior is the complete removal of types. In versions prior to 8.0, many REST APIs supported the concept of a type within an index. 6.x and 7.x deprecated all functionality related to types, so that in 8.0 types will be completely removed. However, many REST API shape elements in 7.x are tightly correlated to the concept of a type. For example, in 7.x you could create an index with a custom type (e.g. PUT foo/custom_type/1 { "a" :"b"}). While that functionality is deprecated in 7.x, it is still supported in 7.x. However in 8.0, how can we still support the type based URLs for Rest API version compatibility if types will be completely removed ?
A form of behavior compatibility will be introduced on a case-by-case basis. Behavior compatibility’s primary focus is to best-effort honor the intent of the request and response and provide shape compatibility. In most cases behavior compatibility will not be identical to the prior version.
For the complete removal of types, the behavior compatibility strategy will be to internally map any type based request to a typeless based request. For example, (if compatibility is requested) the server will still accept typed based request (e.g. PUT foo/custom_type/1 { "a" :"b"}), but will internally convert to a typeless request (e.g. PUT foo/_doc/1 { "a" :"b"}). The goal here is to ensure that the requested shape is still compatible, honoring the intent of the request, but not behave exactly as it did in the prior version.
The behavior compatibility will be best effort to honor the intent of the request and preserve the format of the response. However, this will not always be sufficient for all use cases. In the above example, if the application logic was dependent on custom_type the behavior compatibility would not be sufficient. The service/client will still need to upgrade prior to Elasticsearch for those use-cases. Since compatibility will be requested per HTTP request via a header, clients can opt-out or opt-in per request.
The documentation will be updated to help clearly identify exactly how the behavior compatibility will be achieved to allow consumers better understand if compatibility will work for their use cases.
Deprecations and Compatibility
Elasticsearch follows a pattern of only introducing breaking changes in major release (with very few exceptions). All breaking changes are deprecated in a major version before they are implemented in the next major version. Rest API version compatibility will not change any of the existing deprecation or release strategies. However, Rest API version compatibility will extend the ability to use deprecated API for a full extra major release if (and only if) REST compatibility is requested for that API and version. Also, note that due the the case-by-case of behavior compatibility, the behavior that backs the REST API may not be sufficient for all use cases.
Additionally, a new type of warning will be introduced. A compatibility warning, much like the deprecation warnings will be introduced to help systems/clients understand where and what REST API version compatibility is doing. REST API version compatibility should be viewed as a compatibility layer to help systems/clients decouple their upgrade life-cycles for each other. It is not a long term strategy to keep using deprecated features.
Scope
Rest layer wire compatibility, but not behavior compatibility.
Duration
~ 12 months
List of high level tasks
TODO
Parking lot (things discussed, but need to revisit)
[ ] The relationship between Accept and Content-Type headers with/out compatibility. see #53228 (comment) PR to address this