[Draft] K8s otel overview dashboard

[tetianakravchenko]

Proposed commit message

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Screenshots

Current state:

TODO:

• add total numbers: nodes, namespaces, workloads (? - for later maybe)
  • workload is missing for now
  • add container visualisation :
    • number of restarts
    • unhealthy containers (? maybe)
• cluster-wide - keep colors, but use defaults
• should be tested with multi nodes
• add the same panels we have with ad-hoc thing for deploy/ds/sts/pods
• add workload + nodes events warning + errors
• add grouping
• add filtering panel
• To add a tag : OpenTelemetry sth
• Document somewhere the final dashboard
• Use the api command to import it and test it

decisions:

• not change manifests: pods allocatable and node conditions
• cronjob/jobs are not included in the workload (for now)

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: bc96b68

Failed CI Steps

• :pipeline:⬆️ Upload Pipeline: .buildkite/pipeline.yml

History

• 💔 Build #15201 failed e50ef24
• 💔 Build #15140 failed 6ab7e04
• 💔 Build #15133 failed 8f7dd00
• 💔 Build #15104 failed 6eba660

[ChrsMark]

A couple of comments for alignment with the spec.

1. Could we try to be consistent with the terminology, i.e. usage vs utilization vs pct? Our panel titles etc should be aligned with the semantic convention definitions. Hence we should only use usage or utilization where it applies (no need for pct, utilization is a ratio anyways).
2. Could we consider leveraging the limits' utilization metrics as well? There was strong push-back by the community when we introduced the *.node.utilization metrics so relying only these might be risky in case the community decides to deprecate them. From my perspective these are nice to haves but we can rely to *.cpu.usage ones and the limit based utilizations.
3. Also in the panels it would be nice if we explicitly mention what utilization we display (ratio against the node's capacity VS ratio against the limits).

[ChrsMark]

k8s.node.cpu.usage metric represents the number of cores used in a time window, hence the term Pct is not accurate here. Or there is a specific reason for mentioning this?

[ChrsMark]

Isn't this CPU utilization instead of usage? Also maybe you can consider explicitly mentioning that this utilization is against the Node's capacity and maybe add another graph for the limit based utilizations?

[tetianakravchenko]

All comments were addressed in #11310, closing this PR