Detect Elasticsearch version to handle doc_type appropriately

[spencergilbert]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Based on discussion from #9970 it would be a nicer user experience for Vector to detect Elasticsearch's version and appropriately handle the type name. Example from fluentd

[StephenWakely]

As per the comment here the api is not always authorized. We may need to provide a fallback config option to specify the version.

[ktff]

It's possible to detect the version at startup as mentioned in #13881 (comment), else fallback to configured version, else error during build if version isn't configured.

[jszwedko]

It's possible to detect the version at startup as mentioned in #13881 (comment), else fallback to configured version, else error during build if version isn't configured.

My thought is similar, but slightly different: let users configure whether to use auto detection or a static version like:

api_version: auto | v7 | v8

If they configure auto and the detection fails then Vector should halt. Typically I imagine users will want to configure a specific API version.

[ktff]

So a full proposal would be:

• Add option api_version: auto | v6 | v7 | v8 which can be expanded with v9 | v10 ... as they come. Also this can be expanded with v1 | v2 | v5 if the sink supports them.
• Deprecate suppress_type_name.
• On auto, detect ES version on build. If it fails, return error that can halt Vector. Although this could cause reload to shutdown Vector in one case, considering that it's not common and that it's avoidable by setting exact version this seems to be acceptable.

[jszwedko]

Yep! That is my thought.

[jszwedko]

Re-opening since we had to temporarily revert PR in #15006.