Header parsing for Rest versioning

[pgomulka]

when accessing a compatible API a user has to specify a version its client is compatible with
Some people prefer to use a custom header for that purpose but it is also a common practice on the internet to use Accept and/or Content-Type for that purpose.
The argument against custom header is that it might often be filtered by firewalls or proxies.
This however might also happen with non IANA registered custom values of Accept/Content-Type.
There are several 'patterns' that people follow for values of Accept/Content-Type. Most popular are in a form:
application/vnd.elasticsearch.v7+json - however if we ever decide to register our subtype then we would have to do it for every new version. Also does not indicate well enought that the API is only compatible.

application/vnd.elasticsearch+json; compatibleWith= 7 - this seems to be the most preferable to me, it indicates the intention that the API is not version 7 but only compatible. Also registration would be easier. The subtype will not change - vnd.elasticsearch and we will just define that compatibleWith expects a major version number.

relates #51816

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/REST API)

[bpintea]

Do you refer to the Accept with "Accept-Type"? Asking since I couldn't find any reference to an HTTP Accept-Type header.

Also, wondering about considerations about (or thoughts against) using query strings (or URL params) to signal the desired compatibility, besides using already registered media types in the Accept/Content-Type headers (the content still is "plain" JSON/CBOR etc.). This way the use of HTTP would remain vanilla and thus maybe safe in face of mid-way undesired modifications.

[pgomulka]

@bpintea right Accept - fixed the first comment. Thank you

I think the path param - or any path modification - will be problematic as some users will hardcode permalinks to the api within their code.

[bpintea]

some users will hardcode permalinks to the api within their code.

Wouldn't the code that adds a custom media type in the header be in "proximity" to the code that handles the URL? I'm not in the clear why one would be significantly more difficult than the other, but I admit I don't have a great overview in this regard.

However, not having to mangle the URL and have the header "separation" does feel cleaner. Furthermore I see that RFC6838 does mandate the proposed format so it should be standards-compliant to a certain degree even without the actual IANA registration. (And one can find further endorsement in literature on the topic.)

Which are arguments in favour of the original proposal.

[ezimuel]

We discussed in a meeting with @Mpdreamz and I'm also in favor of having a separate option as compatibleWith instead of a specific subtype. That said, I've some concern about the naming, since compatible sounds almost like act as. If I understood right, this Elasticsearch feature is to format the json output according to a specific version (e.g. 7).
For instance, you are using Elasticsearch 8 with and a client perform a request with compatibleWith=7. Since its is a new major, maybe, the ES engine will respond with a different result, even if the output will be formatted as 7. Here, I see a potential issue for BC break even if the format is ok but the data (the meaning) is different.

My proposal is to use a different naming, for instance format=7 that is more related to the real meaning, supporting specific output format. Moreover, I think we should document this feature carefully, explaining what really means "compatible".

[jaymode]

@pgomulka @jakelandis ping regarding this issue and the comment from @ezimuel. I know that most of the work recently has been using compatible with and IMO I think we should close this as code has landed to do this parsing. Do you see a reason to keep this issue open?