X-Opaque-ID contains UUID causing ES deduplication to fail

[pgomulka]

ES is originally using X-Opaque-ID to trace origin of the request, to help identify slow search request or the use of deprecated functionality.
There was an expectation that this value would contain something in a form of user-id. No specification was given on how this id will look like. We documented this in both slow log and deprecation use cases.

The value of the X-Opaque-ID is used when de-duplicating deprecated log messages (together with a key of the deprecated feature). That means if the X-Opaque-ID is always unique, then there will be no de-duplication of those messages.
This will cause deprecation log to grow and since we enabled by default deprecation log indexing, it might use significant resources of the cluster.

The problem:
Kibana is using the UUID in a form of 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' for the X-Opaque-ID to help diagnose conceptual origin of the HTTP request. From what I understand, it can also pass the value provided by a user on a request to Kibana?
But because the value is mostly unique, it generates a lot of duplicated deprecation logs.

I wonder if we could use trace-id for UUID and use something like a UserID for the X-Opaque-Id

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[pgomulka]

we also discussed this on a core-infra sync today.
We realised that maybe two separate use cases are mixed here. The tracing of the request across the stack and the identification of the origin of the request.
For the tracing we would prefer traceparent/traceid be used. We need to expand the documentation on this. Here is the PR that added the support for this header https://github.com/elastic/elasticsearch/pull/74210/files
The traceparent can have a generated UUIDs (following the defined syntax)

For origin of the slow request/depreacated feature request we would prefer if X-Opaque-ID contain an id from a finite set so that we could deduplicate deprecatation logs. As we documented the intended use case is to set a user-id (no format enforced) so that an app/user could be found by a user.
For Kibana, maybe an X-Opaque-ID could be something like a call-site. A class name, method? and for a traceparent a UUID could be used?

[droberts195]

For Kibana, maybe an X-Opaque-ID could be something like a call-site. A class name, method?

Call-site would be useful, but in some cases it would be nice to also know the object that made the call.

Recently we have had a number of support cases where something in Kibana has been submitting a search to Elasticsearch that uses enormous resources, for example SIEM detection rules. In this case it would be nice to not just find out that the call site is "detection rules", but to find out the name of the particular rule that was involved.

I think this change would be in keeping with the original intent of this issue. For both deprecation log de-duplication and identification of the source of heavy searches it's useless to have a UUID for each search done by a particular SIEM detection rule. And in both cases it's useful to know not just that the source of the search was a SIEM detection rule, but which one.

Maybe simply call-site could be used most of the time, with call-site + object name to be used in a select few cases where we anticipate needing to identify the object.

[lukeelmers]

For origin of the slow request/depreacated feature request we would prefer if X-Opaque-ID contain an id from a finite set so that we could deduplicate deprecate log

I completely understand the need to deduplicate the deprecation logs and the challenges that unique opaque IDs cause, but TBH I'm a little confused on this -- don't we call the ID "opaque" exactly because Elasticsearch shouldn't be making any assumptions about its contents?

What makes this challenging for Kibana is that we are doing more with x-opaque-id than just adding request uuid. We now have an execution context service which will be enabled by default starting in 8.1. The service allows plugins to create a context object that is propagated to ES via the x-opaque-id header.

This means that x-opaque-id can now look something like this (quoting an example from #119932):

Then, in Elasticsearch slow logs you might find a hint that a request was initiated by my-plugin-ui-feature feature of my-app application and proxied through my-plugin-server-feature features of my-app-server app:

x-opaque-id: ${requestId};kibana:my-app:my-plugin-ui-feature:42;my-app-server:my-plugin-server-feature:42

All this is to say, if we wanted to change the value Kibana passes via x-opaque-id, it would also affect our implementation of the execution context service. But I'll defer to @mshustov for his thoughts on that.

[rudolf]

I think it's worth discussing what we want to do long term. But we need to find a solution for 7.16/7.17

I think we have the following options:

1. Remove x-opaque-id from Kibana. Either without any replacement tracing header or by switching to the traceparent header. The Kibana Core can explore the effort and any potential problems that might come out of this solution.
2. When receiving the x-elastic-product-origin: 'kibana' header, Elasticsearch ignores the x-opaque-id header when deduplicating logs. Would this be a viable alternative?

[mshustov]

There was an expectation that this value would contain something in a form of user-id. No specification was given on how this id will look like. We documented this in both slow log and deprecation use cases.

Since this format's been documented for a while, I agree that it's a bug. In #71019 Kibana didn't require x-opaque-id to be a user unique identifier and as it's noticed Kibana falls back to a random uuid value if x-opaque-id is not provided.

We now have an execution context service which will be enabled by default starting in 8.1. The service allows plugins to create a context object that is propagated to ES via the x-opaque-id header.
This means that x-opaque-id can now look something like this

In the long term, we are going to migrate from (mis-)used x-opaque-id header to semantically correct baggage header #101708. The draft status of the baggage header specification is the main blocker for this effort. If it's a critical problem we can consider using another header for execution context propagation.

Remove x-opaque-id from Kibana.

We can't just remove x-opaque-id support as it will be a breaking change. Kibana already documents server.requestId settings https://www.elastic.co/guide/en/kibana/7.16/settings.html However, there are no requirements for x-opaque-id content.

What I'd propose is:

1. document requirements to x-opaque-id content
2. stop generating uuid in Kibana if x-opaque-id is not provided
3. decide what information Kibana should provide via x-opaque-id header.
   @droberts195 suggested Call-site would be useful, but in some cases it would be nice to also know the object that made the call. . We can provide a static kibana value if execution_context is disabled, or a more specific kibana enitity value if execution_context is enabled. execution_context can hint whether a request is originated from a Kibana visualization or Kibana background task/alert. In either way, we get rid of uuid value in x-opaque-id header which should stabilize the deprecation logging. WDYT? @droberts195 @pgomulka

As a side note: we need to document x-opaque-id content format requirement for other Elastic products. For example, other elasticsearch-js client doesn't document this requirement either https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/observability.html#_x_opaque_id_support

[pgomulka]

document requirements to x-opaque-id content
stop generating uuid in Kibana if x-opaque-id is not provided
decide what information Kibana should provide via x-opaque-id header.
@droberts195 suggested Call-site would be useful, but in some cases it would be nice to also know the object that made the call. . We can provide a static kibana value if execution_context is disabled, or a more specific kibana enitity value if execution_context is enabled. execution_context can hint whether a request is originated from a Kibana visualization or Kibana background task/alert. In either way, we get rid of uuid value in x-opaque-id header which should stabilize the deprecation logging. WDYT?

this sounds great to me, because this is exactly what ES would expect on that header at the moment.
I will work on improving the documentation for x-opaque-id and traceparent/trace.id fields

[droberts195]

We can provide a static kibana value if execution_context is disabled, or a more specific kibana enitity value if execution_context is enabled.

I guess this relates to #101587 right? (Sorry, I don't know much about prior work in this area.)

In that case I think what I was asking for is covered, but an important point is that all the sub-teams whose Kibana apps might submit heavy searches to Elasticsearch actually take advantage of the execution_context functionality. It probably needs a publicity drive when it's ready for use to ensure that all the people who work on Kibana apps for security, APM, alerts, ML, etc, are all aware that heavy searches should be submitted with an execution_context that clearly identifies where they came from.

Anyway, I've subscribed to that other issue and I don't think it needs covering any more on this issue.