Include tracing information in the Kibana logs

[mshustov]

Summary

closes #102704
closes #102699
I combined both steps in a single task to simplify testing. The work consists of two parts:

Changing APM defaults

APM nodejs agent needs to instrument Kibana code to track transactions and spans.
By default, the APM agent sends collected metrics to an APM server for ingestion. Not all of ours users want to store APM metrics in the ES index, so the APM agent introduced contextPropagationOnly configuration flag. From the docs:

context_propagation_only is a boolean configuration option to have an APM agent perform
trace-context propagation and log correlation only; and to explicitly not send event data to APM server.
This allows an application to get automatic context propagation and log correlation,
without having deployed an APM server for event collection.

contextPropagationOnly:true disables metrics and calculation of breakdown metrics that should minimize the performance overhead.

Thus, Kibana APM config has been changed to instrument code but it does not send data to the APM server. There is a table with different scenarios and APM config configuration for each of them:

mode	kibana.yml	finalized agent config
APM agent sends data to the APM server, Kibana relies on the agent to implement log correlation.	active: true	{ active: true, contextPropagationOnly: false }
APM agent doesn't send data to the APM server, but Kibana uses it to implement log correlation.	not declared	{ active: true, contextPropagationOnly: true }
APM agent is disabled completely.	active: false, contextPropagationOnly: false	{ active: false, contextPropagationOnly: false }

Emit trace ids into the Kibana logs

Kibana reads span.id, transaction.id and trace.id from the APM agent and emits them into Kibana logs.
Whenever a user enables logs in ECS JSON format, trancing information is emitted as ESC-compatible Tracing Fields https://www.elastic.co/guide/en/ecs/current/ecs-tracing.html
Logs with pattern: layout do not contain this information, we can support it later if necessary.

How to test

• Enable ECS JSON logging:

logging:
  appenders:
    consolejson:
      type: console
      layout:
        type: json
  root:
   appenders: [consolejson, default]
   level: all

• run Kibana
• Make sure trace data are included in the logs. Note that not all the Log records have got them. Currently not instrumented:
  • Async tasks initiated by Kibana server
    • data plugin server-side requests (because of the bsearch implementation?)
    • some task manager operations (Running task...)
  • Kibana server HTTP responses (opened "http.server.response" logging should include tracing information. #114476)

Risk Matrix

When forming the risk matrix, consider some of the following examples and how they may potentially impact the change:

Risk	Probability	Severity	Mitigation/Notes
APM agent adds performance overhead	High	Medium	Wait for nodejs@v16 upgrade or disable instrumentation by default.

Instrumenting runtime code with APM agent with contextPropagationOnly adds a tiny performance overhead. In DemoJourney test scenario, 95th perc. of the response time deviation has the value of statistical error.

master version:

branch version:

Checklist

Delete any items that are not applicable to this PR.

• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios

[trentm]

In DemoJourney test scenario, 95th perc. of response time increased from 15% to 100%.

What row(s) of the posted tables show this?

[mistic]

@elasticmachine merge upstream

[mshustov]

@elasticmachine merge upstream

[kibanamachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 4b025d2
• Storybooks Preview

Metrics [docs]

✅ unchanged

History

• 💚 Build #5697 succeeded bf121b0
• 💚 Build #5678 succeeded 6417b5b
• 💔 Build #4889 failed e2067e5
• 💔 Build #4853 failed bb6251b
• 💔 Build #4609 failed 62dda4f
• 💚 Build #4578 succeeded 95e4199

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

.