provide custom metrics APIService

[siggy]

Summary

Kubernetes provides a Horizontal Pod Autoscaler to enable scaling pods based on metrics. These metrics may come from the container (cpu, mem), or, via a custom metrics APIService.

The autoscaling thanos-demo demonstrates using a k8s-prometheus-adapter to read Linkerd latency metrics, expose them via a custom metrics APIService, and affect scaling via an HPA.

Proposal

Implement a custom metrics API service in Linkerd's control-plane. This would enable autoscaling out-of-the-box, and also enable Linkerd's metrics to be easily consumable via systems that expect Kubernetes metrics.

[grampelberg]

See https://github.com/grampelberg/leaderboard for how to get this working!

[kevingbb]

It would be great to be able to leverage linkerd custom metrics such as rps, success rate and latency without having to install a custom metrics adapter like k9s-prometheus-adapter for example.

The additional challenge I foresee in this is the fact that there can only be a single custom metrics adapter per k8s installation. So if the prometheus adapter has been installed already, or something like keda has already been installed, then a linkerd custom metrics adapter would overwrite both. Or maybe my misunderstanding of the custom and external metrics endpoints in k8s is wrong.

[grampelberg]

then a linkerd custom metrics adapter would overwrite both

You're correct, there can only be one!

[geekyfox90]

The solution is to use Prometheus federation API in order to copy linkerd metrics to the prometheus linked with your prometheus adapter.

Check out this article on how to use the federation API to get linkerd metrics on your prometheus used by the prometheus adapter.
https://linkerd.io/2/tasks/exporting-metrics/

I tested this and it works perfectly :)

[naseemkullah]

The solution is to use Prometheus federation API in order to copy linkerd metrics to the prometheus linked with your prometheus adapter.

Any downside to connecting the prom adapter to the copied metrics prometheus rather than at the source linkerd-prometheus (e.g. less acurate, less precise, less reactive or anything of the sort)?

[grampelberg]

@naseemkullah there'll be a delay and it will be dependent on how you've configured your own prometheus installation.