Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Dataset Quality] Fix failing es-promotion test on 8.x #205188

Merged
merged 5 commits into from
Dec 30, 2024
Merged

[Dataset Quality] Fix failing es-promotion test on 8.x #205188

merged 5 commits into from
Dec 30, 2024

Conversation

awahab07
Copy link
Contributor

@awahab07 awahab07 commented Dec 27, 2024 •
edited by kibanamachine
Loading

Fixes #203956
Related to #203165

Summary

Newer builds of ES ingest event.ingested and tags fields into APM indices making the Dataset Quality tests fail which rely on ingested field counts against set number of field limits. The PR addresses this by explicitly ingesting the fields beforehand.

After the merge to main, the PR will be backported to 8.x to unblock es-promotion.

Example of docs on promoted ES vs current:

promoted EScurrent ES
 
            {
                "_id": "8TTJ9pMBAWfrnWXiHPCS",
                "_ignored": [
                    "cloud",
                    "log.level",
                    "test_field"
                ],
                "_index": ".ds-logs-apm.app.tug-default-2024.12.24-000001",
                "_score": 1,
                "_source": {
                    "@timestamp": 1735011997767,
                    "cloud.project.id": "6906200000000291",
                    "data_stream.dataset": "apm.app.tug",
                    "data_stream.namespace": "default",
                    "data_stream.type": "logs",
                    "event": {
                        "ingested": "2024-12-24T03:49:55Z" # This was making the test fail
                    },
                    "event.dataset": "apm.app.tug",
                    "host.name": "synth-host",
                    "input.type": "logs",
                    "log.level": "<large-value>",
                    "message": "a log message",
                    "network.bytes": 3249,
                    "service.name": "test_service",
                    "tags": [
                        "this_is_here_to_remove_variance_introduced_by_the_geoip_processor",
                        "_geoip_database_unavailable_GeoLite2-City.mmdb"
                    ],
                    "test_field": [
                        "<large-value>"
                    ],
                    "tls.established": true,
                    "trace.id": "6906200000000290"
                }
            }
 
            {
                "_id": "VgHU9pMBiLBjaRQIIxwZ",
                "_ignored": [
                    "cloud",
                    "log.level",
                    "test_field"
                ],
                "_index": ".ds-logs-apm.app.tug-default-2024.12.24-000001",
                "_score": 1,
                "_source": {
                    "@timestamp": 1735012716806,
                    "cloud.project.id": "7497700000000291",
                    "data_stream.dataset": "apm.app.tug",
                    "data_stream.namespace": "default",
                    "data_stream.type": "logs",
                    "event.dataset": "apm.app.tug",
                    "host.name": "synth-host",
                    "input.type": "logs",
                    "log.level": "<large-value>",
                    "message": "a log message",
                    "network.bytes": 4020,
                    "service.name": "test_service",
                    "tags": [
                        "this_is_here_to_remove_variance_introduced_by_the_geoip_processor",
                        "_geoip_database_unavailable_GeoLite2-City.mmdb"
                    ],
                    "test_field": [
                       "<large-value>"
                    ],
                    "tls.established": false,
                    "trace.id": "7497700000000290"
                }
            }

@awahab07
Ingest event.dataset beforehand in field limit tests as new version…
de85439 
… of ES ingests it itself making the tests fail.
@awahab07 awahab07 added release_note:skip Skip the PR/issue when compiling release notes backport:prev-minor Backport to (9.0) the previous minor version (i.e. one version back from main) Team:obs-ux-logs Observability Logs User Experience Team labels Dec 27, 2024
@awahab07 awahab07 requested review from a team as code owners December 27, 2024 10:05
@elasticmachine
Copy link
Contributor

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)

@awahab07 awahab07 requested a review from a team as a code owner December 27, 2024 10:44
@botelastic botelastic bot added ci:project-deploy-observability Create an Observability project Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team labels Dec 27, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

@github-actions GitHub Actions
Copy link
Contributor

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

  • /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

tonyghiani
tonyghiani approved these changes Dec 27, 2024
View reviewed changes
Copy link
Contributor

@tonyghiani tonyghiani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@elasticmachine
Copy link
Contributor

elasticmachine commented Dec 30, 2024 •
edited
Loading

💛 Build succeeded, but was flaky

  • Buildkite Build
  • Commit: 2c26dbc
  • Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-205188-2c26dbcf7106

Failed CI Steps

Metrics [docs]

✅ unchanged

History

@awahab07 awahab07 merged commit bd9254f into elastic:main Dec 30, 2024
8 checks passed
@kibanamachine
Copy link
Contributor

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/12546093134

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Dec 30, 2024
@awahab07
[Dataset Quality] Fix failing es-promotion test on 8.x (elastic#205188
82f86f9 
)

Newer builds of ES ingest `event.ingested` and `tags` fields into APM
indices making the Dataset Quality tests fail which rely on ingested
field counts against set number of field limits. The PR addresses this
by explicitly ingesting the fields beforehand.

After the merge to main, the PR will be backported to `8.x` to unblock
es-promotion.

(cherry picked from commit bd9254f)
@kibanamachine kibanamachine mentioned this pull request Dec 30, 2024
Merged
@kibanamachine
Copy link
Contributor

💚 All backports created successfully

Status Branch Result
✅ 8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

kibanamachine added a commit that referenced this pull request Dec 30, 2024
@kibanamachine @awahab07
[8.x] [Dataset Quality] Fix failing es-promotion test on &#x60;8.x&#x…
d18a51c 
…60; (#205188) (#205246)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Dataset Quality] Fix failing es-promotion test on &#x60;8.x&#x60;
(#205188)](#205188)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Abdul Wahab
Zahid","email":"awahab07@yahoo.com"},"sourceCommit":{"committedDate":"2024-12-30T11:46:49Z","message":"[Dataset
Quality] Fix failing es-promotion test on `8.x` (#205188)\n\nNewer
builds of ES ingest `event.ingested` and `tags` fields into
APM\r\nindices making the Dataset Quality tests fail which rely on
ingested\r\nfield counts against set number of field limits. The PR
addresses this\r\nby explicitly ingesting the fields
beforehand.\r\n\r\nAfter the merge to main, the PR will be backported to
`8.x` to
unblock\r\nes-promotion.","sha":"bd9254fe1ebbc042968846dbd1fd52f1769dc0bb","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:prev-minor","ci:project-deploy-observability","Team:obs-ux-logs","Team:obs-ux-infra_services"],"title":"[Dataset
Quality] Fix failing es-promotion test on
`8.x`","number":205188,"url":"https://github.com/elastic/kibana/pull/205188","mergeCommit":{"message":"[Dataset
Quality] Fix failing es-promotion test on `8.x` (#205188)\n\nNewer
builds of ES ingest `event.ingested` and `tags` fields into
APM\r\nindices making the Dataset Quality tests fail which rely on
ingested\r\nfield counts against set number of field limits. The PR
addresses this\r\nby explicitly ingesting the fields
beforehand.\r\n\r\nAfter the merge to main, the PR will be backported to
`8.x` to
unblock\r\nes-promotion.","sha":"bd9254fe1ebbc042968846dbd1fd52f1769dc0bb"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/205188","number":205188,"mergeCommit":{"message":"[Dataset
Quality] Fix failing es-promotion test on `8.x` (#205188)\n\nNewer
builds of ES ingest `event.ingested` and `tags` fields into
APM\r\nindices making the Dataset Quality tests fail which rely on
ingested\r\nfield counts against set number of field limits. The PR
addresses this\r\nby explicitly ingesting the fields
beforehand.\r\n\r\nAfter the merge to main, the PR will be backported to
`8.x` to
unblock\r\nes-promotion.","sha":"bd9254fe1ebbc042968846dbd1fd52f1769dc0bb"}}]}]
BACKPORT-->

---------

Co-authored-by: Abdul Wahab Zahid <awahab07@yahoo.com>
stratoula pushed a commit to stratoula/kibana that referenced this pull request Jan 2, 2025
@awahab07 @stratoula
[Dataset Quality] Fix failing es-promotion test on 8.x (elastic#205188
94f941a 
)

Newer builds of ES ingest `event.ingested` and `tags` fields into APM
indices making the Dataset Quality tests fail which rely on ingested
field counts against set number of field limits. The PR addresses this
by explicitly ingesting the fields beforehand.

After the merge to main, the PR will be backported to `8.x` to unblock
es-promotion.
benakansara pushed a commit to benakansara/kibana that referenced this pull request Jan 2, 2025
@awahab07 @benakansara
[Dataset Quality] Fix failing es-promotion test on 8.x (elastic#205188
681c9d4 
)

Newer builds of ES ingest `event.ingested` and `tags` fields into APM
indices making the Dataset Quality tests fail which rely on ingested
field counts against set number of field limits. The PR addresses this
by explicitly ingesting the fields beforehand.

After the merge to main, the PR will be backported to `8.x` to unblock
es-promotion.
cqliu1 pushed a commit to cqliu1/kibana that referenced this pull request Jan 2, 2025
@awahab07 @cqliu1
[Dataset Quality] Fix failing es-promotion test on 8.x (elastic#205188
a960223 
)

Newer builds of ES ingest `event.ingested` and `tags` fields into APM
indices making the Dataset Quality tests fail which rely on ingested
field counts against set number of field limits. The PR addresses this
by explicitly ingesting the fields beforehand.

After the merge to main, the PR will be backported to `8.x` to unblock
es-promotion.
CAWilson94 pushed a commit to CAWilson94/kibana that referenced this pull request Jan 13, 2025
@awahab07 @CAWilson94
[Dataset Quality] Fix failing es-promotion test on 8.x (elastic#205188
3db7227 
)

Newer builds of ES ingest `event.ingested` and `tags` fields into APM
indices making the Dataset Quality tests fail which rely on ingested
field counts against set number of field limits. The PR addresses this
by explicitly ingesting the fields beforehand.

After the merge to main, the PR will be backported to `8.x` to unblock
es-promotion.
viduni94 pushed a commit to viduni94/kibana that referenced this pull request Jan 23, 2025
@awahab07 @viduni94
[Dataset Quality] Fix failing es-promotion test on 8.x (elastic#205188
cb2baff 
)

Newer builds of ES ingest `event.ingested` and `tags` fields into APM
indices making the Dataset Quality tests fail which rely on ingested
field counts against set number of field limits. The PR addresses this
by explicitly ingesting the fields beforehand.

After the merge to main, the PR will be backported to `8.x` to unblock
es-promotion.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@crespocarlos crespocarlos crespocarlos approved these changes

@tonyghiani tonyghiani tonyghiani approved these changes

Assignees
No one assigned
Labels
backport:prev-minor Backport to (9.0) the previous minor version (i.e. one version back from main) ci:project-deploy-observability Create an Observability project release_note:skip Skip the PR/issue when compiling release notes Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team Team:obs-ux-logs Observability Logs User Experience Team v8.18.0 v9.0.0
Projects
None yet
Milestone
No milestone
5 participants
@awahab07 @elasticmachine @kibanamachine @crespocarlos @tonyghiani