Kepler general energy metrics vs energy tracing metrics

[williamcaban]

Is your feature request related to a problem? Please describe.

The recent changes in the kind of metrics exposed increase the level of visibility to trace how much energy is consumed by which elements on a process/Pod/container. This kind of metrics is valuable for applications owners looking to optimize the energy consumption by their application or services stack. At the same time, the granularity of these metrics make it extremely difficult for functionalities that require a quick understanding of the system (e.g. schedulers, multi-cluster level dashboards, etc) and considerable increase the load on the Prometheus stack.

Describe the solution you'd like

I like to propose for Kepler to have a flag that can be use to influence the type of metrics that it exposes on the metric endpoint. Basically, enable Kepler to be used by general use cases (e.g. report the top-5 energy consumption by namespace, pods or apps; report the software+OS-view energy consumption by nodes, etc) and another mode for developers that want to do low level tracing of energy consumption of their apps.

The mode for the general use cases should provide a simple single metric per container or Pod, that can be used for user facing reporting and executive types of reports.

Describe alternatives you've considered
With the current granular metrics:

• We have seen a significant impact on Prometheus utilization (way too many metrics in very short time even on micro environments)
• It has become very challenging using the metrics of the general use cases as the query formula gets very long in not time, and the system takes several seconds to process it once
• For comparison, when loading the metrics on DataFrames to run analytics in a Jupyter notebooks for an 8 nodes clusters with ~500 Pods, loading a 2hr dataset with 1 minutes resolution: The current granular metrics takes 2min 17s (as reported by %timeit) to load. With the previous aggregated metrics, the same cluster setup with the same datasets and resolution was loading in less than 1 second.

Additional context
The general use cases should also cover #301

[rootfs]

thank you @williamcaban for the investigation and proposal

• We have seen a significant impact on Prometheus utilization (way too many metrics in very short time even on micro environments)
• For comparison, when loading the metrics on DataFrames to run analytics in a Jupyter notebooks for an 8 nodes clusters with ~500 Pods, loading a 2hr dataset with 1 minutes resolution: The current granular metrics takes 2min 17s (as reported by %timeit) to load. With the previous aggregated metrics, the same cluster setup with the same datasets and resolution was loading in less than 1 second.

This is surely a concern for us. If Prometheus cannot support a small setup, the metrics scheme needs a scalability study.

It has become very challenging using the metrics of the general use cases as the query formula gets very long in not time, and the system takes several seconds to process it once

This can be investigated in two ways: exposing the general use case through metrics or supporting them via Prometheus query. Again, these approaches are to be evaluated via usability and scalability factors.

[rootfs]

cc @marceloamaral

[rootfs]

Let's create a process for metrics schema:

• Support general use cases: current and aggregate power usage by nodes, pods, and potentially namespace.
• Focus on the scalability on evaluating design choices: what would happen to Kepler, Prometheus, or Grafana dashboard when the numbers of pods and nodes increase?
• Add integration test on the quantifying metrics query latency.

@sustainable-computing-io/reviewer

[marceloamaral]

Right, I do agree that we need a metric that sum the energy consumption of all sources:
kepler_container_joules_total as the old pod_curr_energy_millijoule

This metric will represent the container_energy = package+ucore+dram+gpu+other_components

[marceloamaral]

Regarding the number of metrics. We have some ideas of disabling some metrics via configuration.
So we can do that in the future.

However, just keep in mind that node-exporter, cadvisor and other exporters export a lot more metrics than we do without major problems.
But of course we need to do some analysis on this.

[rootfs]

thank @williamcaban for sharing me the setup.
This setup is quite beefy

64 cores Intel(R) Xeon(R) Gold 6338N CPU @ 2.20GHz

It has the latest metrics: kepler_pod_energy.

I ran some quick tests and did find quite some issues:

• current CPU usage is way too high, it is over 150%
  
• latency is also high, 4s vs 1s before refactoring
  

[marceloamaral]

@williamcaban regarding the large prometheus metric being too expensive to process, I created a new metric in PR #370

[marceloamaral]

@rootfs definitely there is something wrong with this deployment.
First, why it has 3 kepler instances?

In my environment the kepler CPU utilization is not high....

[rootfs]

the top output was from a remote tmux console, there are issues with rendering. This is the top from inside the kepler pod

[marceloamaral]

The empty cgroup values might be related to this fix (PR#377)

[rootfs]

@williamcaban @marceloamaral Here is my controlled test results.
https://gist.github.com/rootfs/2bb2c742c3afd9acca7811ed8eba7dd7

The TL;DR is, there is a scalability regression in the current code, kepler's CPU usage is very sensitive to the number of Pods.