Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Cloud Security] [Agentless] Improving error log metadata and sending APM trace id #192235

Merged
merged 13 commits into from
Sep 24, 2024

Conversation

opauloh
Copy link
Contributor

@opauloh opauloh commented Sep 6, 2024

Summary

This PR includes a few improvements in the communication between Kibana and the Agentless API.

  • Adding a X-Request-ID Header on all HTTP calls from the Kibana server to the Agentless API. X-Request-ID is the currentTraceparent string captured from Apm Service and is unique per request.
  • Also, this PR enhances some error logs metadata with the relevant fields, and all documents logged to ES also includes the trace.id field
  • Also this PR redacts the fleet token sent to the debug logs to prevent credential leaking

@opauloh opauloh added Team:Cloud Security Cloud Security team related ci:cloud-deploy Create or update a Cloud deployment labels Sep 6, 2024
@opauloh
Copy link
Contributor Author

opauloh commented Sep 6, 2024

/ci

@opauloh
Copy link
Contributor Author

opauloh commented Sep 6, 2024

/ci

@opauloh opauloh added the ci:project-deploy-security Create a Security Serverless Project label Sep 6, 2024
@opauloh
Copy link
Contributor Author

opauloh commented Sep 6, 2024

/ci

@opauloh opauloh added the ci:cloud-redeploy Always create a new Cloud deployment label Sep 9, 2024
@opauloh
Copy link
Contributor Author

opauloh commented Sep 9, 2024

/ci

@opauloh
Copy link
Contributor Author

opauloh commented Sep 13, 2024

/ci

1 similar comment
@opauloh
Copy link
Contributor Author

opauloh commented Sep 13, 2024

/ci

@opauloh opauloh linked an issue Sep 13, 2024 that may be closed by this pull request
6 tasks
@opauloh
Copy link
Contributor Author

opauloh commented Sep 14, 2024

/ci

@opauloh
Copy link
Contributor Author

opauloh commented Sep 16, 2024

/ci

@opauloh
Copy link
Contributor Author

opauloh commented Sep 20, 2024

/ci

@opauloh
Copy link
Contributor Author

opauloh commented Sep 20, 2024

/ci

@opauloh opauloh added backport:version Backport to applied version labels and removed ci:build-all-platforms ci:build-cloud-image ci:cloud-deploy Create or update a Cloud deployment ci:cloud-redeploy Always create a new Cloud deployment ci:project-deploy-security Create a Security Serverless Project labels Sep 20, 2024
@opauloh opauloh marked this pull request as ready for review September 20, 2024 19:13
@opauloh opauloh requested a review from a team as a code owner September 20, 2024 19:13
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-cloud-security-posture (Team:Cloud Security)

@opauloh opauloh added the release_note:skip Skip the PR/issue when compiling release notes label Sep 20, 2024
@opauloh
Copy link
Contributor Author

opauloh commented Sep 20, 2024

/ci

Copy link
Member

@nchaulet nchaulet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code LGTM 🚀

@botelastic botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Sep 23, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Team:Fleet)

@kibana-ci
Copy link
Collaborator

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #70 / console app misc console behavior customizable font size should allow the font size to be customized

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@opauloh opauloh merged commit 10bcc62 into elastic:main Sep 24, 2024
24 checks passed
@opauloh opauloh deleted the techdebt/agentless_trace_id branch September 24, 2024 16:56
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Sep 24, 2024
… APM trace id (elastic#192235)

## Summary

This PR includes a few improvements in the communication between Kibana
and the Agentless API.

- Adding a `X-Request-ID` Header on all HTTP calls from the Kibana
server to the Agentless API. X-Request-ID is the
[currentTraceparent](https://www.elastic.co/guide/en/apm/agent/nodejs/current/agent-api.html#apm-current-traceparent)
string captured from Apm Service and is unique per request.
- Also, this PR enhances some error logs metadata with the relevant
fields, and all documents logged to ES also includes the
[trace.id](https://www.elastic.co/guide/en/ecs/8.11/ecs-tracing.html)
field
- Also this PR redacts the fleet token sent to the debug logs to prevent
credential leaking

(cherry picked from commit 10bcc62)
@kibanamachine
Copy link
Contributor

💚 All backports created successfully

Status Branch Result
8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

kibanamachine added a commit that referenced this pull request Sep 24, 2024
…ending APM trace id (#192235) (#193901)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Cloud Security] [Agentless] Improving error log metadata and sending
APM trace id (#192235)](#192235)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Paulo
Silva","email":"paulo.henrique@elastic.co"},"sourceCommit":{"committedDate":"2024-09-24T16:56:03Z","message":"[Cloud
Security] [Agentless] Improving error log metadata and sending APM trace
id (#192235)\n\n## Summary\r\n\r\nThis PR includes a few improvements in
the communication between Kibana\r\nand the Agentless API.\r\n\r\n-
Adding a `X-Request-ID` Header on all HTTP calls from the
Kibana\r\nserver to the Agentless API. X-Request-ID is
the\r\n[currentTraceparent](https://www.elastic.co/guide/en/apm/agent/nodejs/current/agent-api.html#apm-current-traceparent)\r\nstring
captured from Apm Service and is unique per request.\r\n- Also, this PR
enhances some error logs metadata with the relevant\r\nfields, and all
documents logged to ES also includes
the\r\n[trace.id](https://www.elastic.co/guide/en/ecs/8.11/ecs-tracing.html)\r\nfield\r\n-
Also this PR redacts the fleet token sent to the debug logs to
prevent\r\ncredential
leaking","sha":"10bcc626ecb1d8bf77f5a4d95d4d85f5e094d782","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Fleet","v9.0.0","Team:Cloud
Security","v8.16.0","backport:version"],"title":"[Cloud Security]
[Agentless] Improving error log metadata and sending APM trace
id","number":192235,"url":"https://github.com/elastic/kibana/pull/192235","mergeCommit":{"message":"[Cloud
Security] [Agentless] Improving error log metadata and sending APM trace
id (#192235)\n\n## Summary\r\n\r\nThis PR includes a few improvements in
the communication between Kibana\r\nand the Agentless API.\r\n\r\n-
Adding a `X-Request-ID` Header on all HTTP calls from the
Kibana\r\nserver to the Agentless API. X-Request-ID is
the\r\n[currentTraceparent](https://www.elastic.co/guide/en/apm/agent/nodejs/current/agent-api.html#apm-current-traceparent)\r\nstring
captured from Apm Service and is unique per request.\r\n- Also, this PR
enhances some error logs metadata with the relevant\r\nfields, and all
documents logged to ES also includes
the\r\n[trace.id](https://www.elastic.co/guide/en/ecs/8.11/ecs-tracing.html)\r\nfield\r\n-
Also this PR redacts the fleet token sent to the debug logs to
prevent\r\ncredential
leaking","sha":"10bcc626ecb1d8bf77f5a4d95d4d85f5e094d782"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/192235","number":192235,"mergeCommit":{"message":"[Cloud
Security] [Agentless] Improving error log metadata and sending APM trace
id (#192235)\n\n## Summary\r\n\r\nThis PR includes a few improvements in
the communication between Kibana\r\nand the Agentless API.\r\n\r\n-
Adding a `X-Request-ID` Header on all HTTP calls from the
Kibana\r\nserver to the Agentless API. X-Request-ID is
the\r\n[currentTraceparent](https://www.elastic.co/guide/en/apm/agent/nodejs/current/agent-api.html#apm-current-traceparent)\r\nstring
captured from Apm Service and is unique per request.\r\n- Also, this PR
enhances some error logs metadata with the relevant\r\nfields, and all
documents logged to ES also includes
the\r\n[trace.id](https://www.elastic.co/guide/en/ecs/8.11/ecs-tracing.html)\r\nfield\r\n-
Also this PR redacts the fleet token sent to the debug logs to
prevent\r\ncredential
leaking","sha":"10bcc626ecb1d8bf77f5a4d95d4d85f5e094d782"}},{"branch":"8.x","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Paulo Silva <paulo.henrique@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:version Backport to applied version labels release_note:skip Skip the PR/issue when compiling release notes Team:Cloud Security Cloud Security team related Team:Fleet Team label for Observability Data Collection Fleet team v8.16.0 v9.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Include APM Trace Id in the X-Request-ID header in all Agentless API requests
5 participants