implementation plan for kubernetes observation

[TaiseiYokoshima]

Current understanding of the general structure of cardamon

1. parses the command line args and load the configuration file
2. builds an execution plan
3. start all startup processes of all scenarios in the given observation
4. execute scenarios one by one

When executing a scenario, for each iteration:

1. start the logging tokio task for processes to observe (both baremetal processes and docker containers)
2. run the iteration of a scenario
3. save the metrics to database
4. continue to the next iteration

For logging, it organises baremetal processes and docker containers into separate vecs and for each vec, a tokio task is spawned up to continuously iterate the processes or containers to query for metrics until tokio token is used to cancel both them. these both aggregate instances of "CpuMetrics" which are processes later.

[TaiseiYokoshima]

Implementation plan

I believe you can separate this into mainly four parts:

• decide on entry structure and its syntax for a process entry of process.type = "kubernetes" for the toml config file. what key and values are allowed. and implement utility functions to handle the config file for kubernetes.
• implement the metrics_logger for kubernetes at src/metrics_logger/kubernetes.rs
• adjust the start_logging function in src/metrics_logger.rs to also include kubernetes logger so that it spawns its tokio task for logging
• research how to get cpu utilisation for kubernetes to generate CpuMetrics

For the kubernetes cpu utilisation queries

I am thinking of using the kube rust crate. Further research is required to know how but this seemed to be the easiest and cleanest to implement.

Other

For kubernetes you can collect metrics per deployment, meaning: query and sum the cpu utilisation of all pods spawned by the given deployment.

need further discussion to support other types of kubernetes resources.

Extra notes

I have understood the bare minimum to write this issue. So far I have understood that the loggers push and aggregate "CpuMetrics" to a type called "MetricsLog" which is essentially a vec of CpuMetrics. I need to understand how the config file is loaded and handled. But, I have understood the general flow of execution.

[ohuu]

This seems like a reasonable plan. I'd like to know more about how you intend to collect the metrics from kubernetes.

I think we should connect to kubernetes directly from cardamon, if that is possible, rather than requiring something like cadvisor.

Maybe the next step should be a small experiment to see what's involved with getting metrics out of kubernetes. I'm not completely opposed to using cadvisor, or something similar, but I'd like to explore if we can do this from within cardamon so that users aren't required to make changes to their software.

[TaiseiYokoshima]

From my understanding, there are 2 options of interfacing with k8s regarding metrics:

1. cAdvisor
2. metrics server

cAdvisor runs per node and collects metrics per container and we would have to interface with the kubelet apis and not the k8s api server.

metrics server is an add-on we can install and it internally uses cAdvisor to collect metrics and provide you with the aggregated metrics per node or per pod. We can retrieve the metrics through the k8s api server.

Do you want to use cAdvisor or the metrics server?

• If you want to use metrics server then this is something that has to be installed which either adds to the requirement to use cardamon for k8s or have some logic that automatically install it and start it up when cardamon runs.
• If you want to go with the cAdvisor route, I have to write code to interface with kubelets apis and aggregate the metrics

Regarding the kube-rs crate, it is a high level crate that makes api requests to the api server for you. If you don' t want me to use kube-rs, I will be using the k8s-openai crate which is just a collection of struct, enum, and type definitions which maps to json payloads of requests and responses btw the k8s api server.

[ohuu]

From what I understand, kube-rs is a rust interface to the k8s API which contains the metrics API. Can't we just connect to the metrics API and get the data directly from there? Isn't that way cadvisor is doing?

[ohuu]

Yeah, reading around this. It's possible to use the standard kubernetes metrics API to get everything we need. No need to install cadvisor or metrics server or anything. Under the hood k8s metrics api actually uses an embedded cadvisor, but that is completely hidden from the user. It doesn't require the user to install anything or change their config.

Basically cardamon would become a kind of metrics server. It would gather all the metrics required to do it's calculation via the k8s metrics API (exposed to us in kube-rs) and the k8s deployment would be completely unchanged and unaware that cardamon is monitoring it.

https://www.kubeblog.com/observability/understanding-how-kubernetes-exposes-metrics/

[TaiseiYokoshima]

Could I assume that cardamon consumes cpu metrics as utilisation in percentage of the cpu on board (all cores)?

[TaiseiYokoshima]

Moreover, I have been reading on k8s APIs and the article you have referenced. My understanding is slightly different from yours or maybe there is a misunderstanding on my part and I would love to clarify.

I am confused between 2 things when you say "k8s metrics API" or "metrics API".

1. The Metadata APIs within the core k8s APIs in the HorizontalPodAutoscaler v2 autoscaling api link to reference
2. The general Metrics API defined outside of the core k8s API but is standardised. reference

If what you meant is the first in the listing above, then I am not sure if those are the kinds of APIs you want to use because those APIs fundamentally rely on HPA to be running. If we use these, that means cardamon cannot measure for deployments or pods that do not use HPA. Moreover, we will be querying average cpu usage per pod.

If what you meant is the second in the listing above, I think this so called "Metics API server" is something you have to implement or bring in as an external dependency like the metrics-server. It is not part of the k8s default bundling or core infrastructure. "Metrics API" is standardised definitions or conventionalised APIs used in k8s. It is not part of the core APIs. So we would have to install and run a dedicated external component like the metrics-server to make those API requests.

I do not know which one you would want me to implement here... Could you let me know?

[ohuu]

Ultimately I'm interested in a solution that places the least burden on the developers using Cardamon. I'm not sure which of these approaches best achieves that. Ideally I would like cardamon to connect to k8s and stream or poll metrics without the user doing anything, but they may not be possible.

I know cadvisor can be used to get metrics from k8s (I've done it before) but that requires adding cadvisor to your deployment. I'm sure there are other methods too.

I think the best thing to do is for you to try one solution and see how it works. We can compare it with another approach later and see which works best.

[TaiseiYokoshima]

I have been experimenting with the direct node api query approach.

currently using the endpoint:
node/proxy/stats/summary
which gives you json object containing a list of pods.
The shape of "pod" is like:

Pod = {
  name: "xxx",
  uid: "xxx",
  cpu: xxx, 
  containers: [
    {
      name: "xxx",
      uid: "xxx",
      cpu: xxx,
    },
    { ... },
    { ... },
    ...

  ]
}. 

The problem with this is that this api will be deprecated soon or is in the works of becoming so:

• Move away from kubelet stats/summary elastic/beats#12792
• Reduce the set of metrics exposed by the kubelet kubernetes/kubernetes#68522

The other option is to use kubelet api:
node/proxy/metrics/resource

which gives you prometheus metric:

container_cpu_usage_seconds_total{container="coredns",namespace="kube-system",pod="coredns-7448499f4d-wp7kr"} 239.636829 1758039279520
container_cpu_usage_seconds_total{container="lb-port-443",namespace="kube-system",pod="svclb-traefik-mb8jp"} 0.013671 1758039278046
container_cpu_usage_seconds_total{container="lb-port-80",namespace="kube-system",pod="svclb-traefik-mb8jp"} 0.014255 1758039278062
container_cpu_usage_seconds_total{container="stress",namespace="default",pod="stress-test-cf5698d66-m7vjs"} 8302.452889 1758039265543
container_cpu_usage_seconds_total{container="stress",namespace="default",pod="stress-test-cf5698d66-sxl4z"} 8309.39079 1758039275773
container_cpu_usage_seconds_total{container="traefik",namespace="kube-system",pod="traefik-6b84f7cbc-v2bq4"} 107.42561 1758039269555


pod_cpu_usage_seconds_total{namespace="default",pod="stress-test-cf5698d66-m7vjs"} 8307.082851 1758039274687
pod_cpu_usage_seconds_total{namespace="default",pod="stress-test-cf5698d66-sxl4z"} 8304.62264 1758039266173
pod_cpu_usage_seconds_total{namespace="kube-system",pod="coredns-7448499f4d-wp7kr"} 239.642238 1758039269198
pod_cpu_usage_seconds_total{namespace="kube-system",pod="svclb-traefik-mb8jp"} 0.04831 1758039253853
pod_cpu_usage_seconds_total{namespace="kube-system",pod="traefik-6b84f7cbc-v2bq4"} 107.449616 1758039274159

With this api, we have to take time stamps together with the cumulative cpu usage for pods and containers. We can either compute the rate while we collect the metrics or compute them as part of the post processing.

This api and similar ones are just cAdvisor embedded in the kubelet and exposed by the kubelet apis.

With this approach cardamon doesn't have to modify any resource and has no side affects to the cluster.

Nonetheless, endpoints like the above ("node/proxy/metrics/resource") is not guaranteed or standardised by k8s. They might give you different format or might not exist at all. So the safest and the most reliable way it to deploy cAdvisor and give access to the container runtime to scrape metrics. cAdvisor scrape metrics from the container runtime via CRI which is standardised and guaranteed by k8s. We just have to ship the right version. Though this might add side effects if not careful. Also, this requires privileged access to connect cAdvisor to the container runtime which users might not want to give to third party tools, because such privilege has the access to destroy the entire cluster.

In regards to the metric server approach

I believe this approach is not favourable. If cardamon were to deploy metrics-server to collect metrics it might have weird side effects. Consider the below:

• what if the user already has a metrics server running? do we override it?
• what if the user wants to measure the initialisation or destruction process of the metrics server itself? cardamon will fail to collect metrics during recording

So to summarise, relying on the metrics server seems too fragile and makes the use of cardamon less flexible and potentially buggy. For example, the user might replace the metrics server during recording which will pause metric collection momentarily.

We should go with the kubelet, or cadvisor approach and the cardamon can abstract out the usage of kubelet or the cadvisor in its config file or some logic to automatically detect it kubelet api availability.