Write deprecation logs to an index

[jasontedor]

The Kibana Upgrade Assistant helps users prepare to the next major version of Elasticsearch. The upgrade assistant works by introspecting various aspects of a cluster and its usage to surface deprecated functionality that is in use, and also to prepare indices that require re-indexing. The deprecations that are surfaced in the migration assistant are those that can be ascertained by introspecting the current state of the cluster. It can not however, catch on-going uses of deprecated functionality (e.g., APIs) that can not be ascertained by introspecting the current state of the cluster. The upgrade assistant would be even more useful for our users if we could assist users in understanding their use of such deprecated functionality.

Today we do surface such deprecated functionality via the deprecation logs. The upgrade assistant does not have an easy way to get its hands on the deprecation logs.

The crux of this issue then is aimed at making it possible for the upgrade assistant to collect the deprecation logs from each running node. One way to do this is to write the deprecation logs to an index that the upgrade assistant could then read, along side the deprecations that it can already obtain.

It is likely that we want to consider the deprecation indices as system indices, and also manage them via ILM if ILM is available (e.g., use of a deprecated API no more recent than N months ago isn't relevant, it's likely that the user already migrated away from that API).

[elasticmachine]

Pinging @elastic/es-core-infra

[elasticmachine]

Pinging @elastic/es-core-features

[pgomulka]

This is a really cool idea. We were looking into this together with @ycombinator and @nik9000 to possibly start a filebeat that would consume ES logs and upload them back to an index.
The solution was proposed a while back here https://github.com/elastic/dev/issues/731

[jaymode]

We discussed this in a couple of team meetings. Two approaches were discussed with the first being that we would rely on a bundled filebeat to ingest the deprecation logs and the other would include building this functionality into the DeprecationLogger itself.

There is ongoing discussion about packaging filebeat and metricbeat with elasticsearch (#49399) but they would be disabled by default. Given the desire to have them be disabled and not add a new process by default, this would mean this functionality would explicitly require the user to do work. Whereas if we added this to the deprecation logger, we could enable it more easily. Given this, the discussion favored the process internal solution.

There was additional discussion regarding:

• Handling rejections on indexing
  • It would be ok to drop messages in this case since this is not critical and retries could cause the cluster/node to get in an even worse state.
• Would indexing these messages add too much load?
  • This should not be the case if we use deprecateAndMaybeLog correctly, which should prevent spamming.
  • However this deduplication currently has no timeout and then we could lose the fact that a deprecated item is being used after the initial document was indexed.
  • Should we use X-Opaque-Id in the deduplication to help the user determine where the deprecated usage is coming from? See Provide more information about cause of deprecation log messages #26836 (comment)
  • For some items like scripting the same deprecated use will result in multiple messages since these could be at the shard level. This should be ok since the majority will not be at that level.