Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Stack Monitoring] Switch cgroup memory fields to keyword #32197

Merged

Conversation

matschaffer
Copy link
Contributor

@matschaffer matschaffer commented Jul 5, 2022

What does this PR do?

Changes cgroup memory fields from long to string.

Per https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html they should be strings. data.go appears to also consider them strings, so just updating the field definitions might be enough.

Why is it important?

With the current mapping as long, ingest of elasticsearch node data can fail when elasticsearch responds with a string like 'max'.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works (used existing suite)
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

So far I can't figure out how to get ES to respond with max in the cgroup memory setting:

$ curl -s -k -u elastic:'(password)' https://localhost:9200/_nodes/stats/os | jq '.nodes | values | .[].os.cgroup.memory'
{
  "control_group": "/",
  "limit_in_bytes": "9223372036854771712",
  "usage_in_bytes": "17973219328"
}

I'll need that before I can advise on how to test this.

Related issues

Use cases

Stack Monitoring

Logs

Without this change you can get this error on ingest:

failed to parse field [elasticsearch.node.stats.os.cgroup.memory.limit.bytes] of type [long] in document with id '5NvEroEBXQqSgbZj8wR7'. Preview of field's value: 'max'\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"For input string: \\\"max\\\"\"}}, dropping event!","service.name":"metricbeat","ecs.version":"1.6.0"}

(from forum)

Per https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html they should be strings. data.go appears to also consider them strings, so just updating the field definitions might be enough.
@matschaffer matschaffer added bug Feature:Stack Monitoring Team:Infra Monitoring UI - DEPRECATED Infrastructure Monitoring UI team - DEPRECATED - Use Team:Monitoring v8.4.0 v8.3.1 labels Jul 5, 2022
@matschaffer matschaffer requested a review from a team as a code owner July 5, 2022 08:03
@matschaffer matschaffer self-assigned this Jul 5, 2022
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jul 5, 2022
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jul 5, 2022

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-07-11T07:46:54.075+0000

  • Duration: 55 min 27 sec

Test stats 🧪

Test Results
Failed 0
Passed 3570
Skipped 884
Total 4454

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@matschaffer matschaffer added backport-v8.3.0 Automated backport with mergify and removed v8.4.0 v8.3.1 labels Jul 7, 2022
@matschaffer matschaffer merged commit 1f2895e into elastic:main Jul 11, 2022
mergify bot pushed a commit that referenced this pull request Jul 11, 2022
* [Stack Monitoring] Switch cgroup memory fields to keyword

Per https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html they should be strings. data.go appears to also consider them strings, so just updating the field definitions might be enough.

* make update

* Update changelog

(cherry picked from commit 1f2895e)

# Conflicts:
#	metricbeat/module/elasticsearch/fields.go
matschaffer added a commit that referenced this pull request Jul 13, 2022
… to keyword (#32295)

* [Stack Monitoring] Switch cgroup memory fields to keyword (#32197)

* [Stack Monitoring] Switch cgroup memory fields to keyword

Per https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html they should be strings. data.go appears to also consider them strings, so just updating the field definitions might be enough.

* make update

* Update changelog

(cherry picked from commit 1f2895e)

# Conflicts:
#	metricbeat/module/elasticsearch/fields.go

* make update

* Remove unrelated changelog entries

* Put trailing lines back

Co-authored-by: Mat Schaffer <mat@elastic.co>
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
… fields to keyword (elastic#32295)

* [Stack Monitoring] Switch cgroup memory fields to keyword (elastic#32197)

* [Stack Monitoring] Switch cgroup memory fields to keyword

Per https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html they should be strings. data.go appears to also consider them strings, so just updating the field definitions might be enough.

* make update

* Update changelog

(cherry picked from commit 05417e4)

# Conflicts:
#	metricbeat/module/elasticsearch/fields.go

* make update

* Remove unrelated changelog entries

* Put trailing lines back

Co-authored-by: Mat Schaffer <mat@elastic.co>
chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023
* [Stack Monitoring] Switch cgroup memory fields to keyword

Per https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html they should be strings. data.go appears to also consider them strings, so just updating the field definitions might be enough.

* make update

* Update changelog
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-v8.3.0 Automated backport with mergify bug Feature:Stack Monitoring Team:Infra Monitoring UI - DEPRECATED Infrastructure Monitoring UI team - DEPRECATED - Use Team:Monitoring
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Stack Monitoring] Mapping for elasticsearch.node.stats.os.cgroup.memory.limit.bytes is incorrect
3 participants