[Stack Monitoring] Switch cgroup memory fields to keyword

[matschaffer]

Rel elastic/beats#31765

Per https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html they should be strings. The internal collection template (monitoring-es.json) has them as keywords without any ignore_above.

This template uses ignore_above on all keywords, so keeping that convention for these mapping updates.

Opening as draft since we won't want to merge this until there's a corresponding beats change in place to match.

Testing

So far I can't figure out how to get ES to respond with max in the cgroup memory setting:

$ curl -s -k -u elastic:'(password)' https://localhost:9200/_nodes/stats/os | jq '.nodes | values | .[].os.cgroup.memory'
{
  "control_group": "/",
  "limit_in_bytes": "9223372036854771712",
  "usage_in_bytes": "17973219328"
}

I'll need that before I can advise on how to test this.

[matschaffer]

elastic/beats#32197 is open, considering this ready for review.

[miltonhultgren]

This means we won't be able to run metric aggregations on this field anymore, is that okay?

[matschaffer]

Good question. I should at least check that we're not trying to graph it in stack monitoring. ES returns a string (potentially max) so if we needed to keep it graphable I guess we could have metricbeat just drop the field if it's not a number.

[matschaffer]

I don't see it in https://github.com/elastic/kibana/blob/b8c1323d5bdc921a6238003bc698bc03750c5320/x-pack/plugins/monitoring/server/lib/metrics/elasticsearch/metrics.ts

Additionally it's already keyword in internal collection so I think we're safe here. Would it be nice to aggregate cgroup memory, perhaps. But I'd probably opt to tackle that as a new feature.

[elasticmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[elasticsearchmachine]

Hi @matschaffer, I've created a changelog YAML for you.

[matschaffer]

Guessing I can probably merge this but maybe someone from @elastic/es-data-management should weigh in too. Also maybe they have ideas on how to get ES producing the max value seen in forum posts.

[jbaiera]

The change looks good to me.

To weigh in on the cgroups question about why "max" appears as a value: The code that collects the cgroup memory values is just reading a single line from either the /sys/fs/cgroup/<GROUP>/memory.limit_in_bytes file (cgroups v1) or the /sys/fs/cgroup/<GROUP>/memory.max file (cgroups v2).

In cgroups v1, a value of -1 indicates that there is no limit on a cgroup's memory, but in v2 a value of max indicates that there is no limit. The value max is default in cgroups v2.

Incidentally, the api that returns these values originally returned a string not because max is a valid value (though it is now) but because some linux distros would return the maximum value of an unsigned 64 bit long if there was no limit. That would overflow a signed long in Java and thus a string was returned instead.

[matschaffer]

Ah, so 9223372036854771712 is max when running on docker for mac I'd guess. Interesting. Thanks @jbaiera !

[matschaffer]

Finally got the metricbeat suite to run against a docker image with this PR (./gradlew buildDockerImage + a follow-up PR to metricbeat mage setup). I'd say we're good to go.

[elasticsearchmachine]

💚 Backport successful

Status	Branch	Result
✅	8.3