Kubernetes Prometheus Proxy is an easy way to get prometheus metrics from application pods running in kubernetes cluster.
In kubernetes, applications are deployed in the form of pods and often there are multiple pods running for application in order to scale horizontally. The application is accessible using ingress routers which directs the request to one of the application pod.
Prometheus is monitoring toolkit which follows pull approach to get the metrics from applications it is monitoring. It makes http call to the configured application target and stores the metrics that it receives in response. Prometheus server has kubernetes service discovery abilities, which can talk to kubernetes api server and generate the list of target pod IPs it wants to scrape based on the configuration. However, for this to work, prometheus server should be able to reach the target pod ip. It can only do that if it is running inside kubernetes cluster.
What if your prometheus server is running externally and cannot reach the target pod ip? Kubernetes Prometheus proxy is helpful proxy which allows pulling metrics from application pods running inside kubernetes cluster. If kubernetes api servers are accessible from outside the kubernetes cluster, you can also configure kubernetes service discovery and pull metrics from pods with the help of this proxy using prometheus's powerful relabel configs.
To pull metrics from all the pods use below http url
http(s)://<prometheus-proxy-hostname>:<port>/mproxy/<kubernetes namespace>/<podname prefix>/<metrics uri>?upstreamPort=<metrics port>&ssl=<true|false>
Where,
prometheus-proxy-hostname : hostname or domain name of prometheus proxy which is accessible externally
port : port on which prometheus proxy is accessible externally
podname prefix: the string with which your application pod names start with
metrics uri: uri of the application which returns metrics
upstreamPort: port on which application pods is exposing metrics on
ssl : if application pod is listening on https then set to true else false. (default false)
Implementation logic
- Check if the metrics path is whitelisted path by matching it with regular expressions defined in METRICS_PATH_WHITELIST environment variable, if it does not match then return HTTP 500
- Query kubernetes master to fetch all pods in the given kubernetes namespace
- Filter pods which are in running state and whose name starts with podname prefix specified in url
- Check if the pod IP address is within CIDR ranges defined in CIDR_WHITELIST environment variable. If it is not then ignore that pod IP
- Query each pod individually using
http(s)://<pod ip>:<upstreamPort>/<metrics uri>(It uses https if ssl=true) - Loop through each metric and add podip and podname as labels to metrics
- Write all these updated metrics onto response
- Close response when all pods have been scraped and their updated metrics are written on response
Example prometheus config
- job_name: example-job
params:
upstreamPort:
- "5043"
ssl:
- "true"
metrics_path: /mproxy/project1/apple/ctxt/metrics
scheme: https
static_configs:
- targets:
- <prometheus-proxy-hostname:port>
When kubernetes api server can be accessed from prometheus server running external to cluster, we can setup kubernetes service discovery and use proxy to pull metric from a pod. Same whitelist checks are performed for checking IP address in whitelisted CIDR and metrics URI matches one of the regex defined in METRICS_PATH_WHITELIST before making http call to get the metrics. For this job config will look like below
- job_name: <name of job>
scheme: <http|https for kubernetes api server>
params:
target_scheme: [<http|https>]
target_port: [<metrics port>]
tls_config:
insecure_skip_verify: true
metrics_path: /kubesd/<metrics url>
kubernetes_sd_configs:
- api_server: <kubernetes api server>
role: pod
bearer_token_file: <location of the token file>
namespaces:
names:
- <kubernetes namespaces>
tls_config:
insecure_skip_verify: true
relabel_configs:
- source_labels: [__meta_kubernetes_pod_ip]
target_label: __param_pod
- source_labels: [__meta_kubernetes_pod_name]
target_label: podname
- source_labels: [__meta_kubernetes_pod_ip]
target_label: podip
- target_label: __address__
replacement: <prometheus proxy hostname>
- source_labels: [__meta_kubernetes_pod_name]
target_label: instance
In addition to above, you can use further relabel_configs to keep or drop pods to be used as targets. For example,
- source_labels: [__meta_kubernetes_pod_name]
regex: .*myapp1.*
action: keep
- source_labels: [__meta_kubernetes_pod_container_name]
regex: deployment
action: drop
In above example, it will keep the pods whose pod name contains myapp1 text, and it will drop pod whose container name contains deployment text.
| Environment Variable | Default Value | Required | Description |
|---|---|---|---|
| K8S_GLOBAL_TOKEN | true | Bearer token used to query kubernetes master to get pod | |
| CIDR_WHITELIST | true | comma separated CIDR list, the pod ip addresses should be within this defined ranges of CIDR | |
| METRICS_PATH_WHITELIST | true | comma separated metrics path regex whitelist. the metrics uri will be validated against these regex expressions and only matching URI will be queried for metrics | |
| TOKEN_FILE | /var/run/secrets/kubernetes.io/serviceaccount/token | false | Absolute path of the file where bearer token is read from. If K8S_GLOBAL_TOKEN is defined then it takes precedence over TOKEN_FILE |
| K8S_CACERT | /var/run/secrets/kubernetes.io/serviceaccount/ca.crt | false | Absolute path of CA certificate file. Used to talk to kubernetes master |
| K8S_NODEJS_PORT | 5050 for http, 5053 for https | false | Port on which app listens on |
| HTTP_PORT | 5050 | false | If certificate environment variables are not defined then http server is started. It will listen on this port. If K8S_NODEJS_PORT is defined then that takes precedence |
| HTTPS_PORT | 5053 | false | If certificate environment variables are defined then https server is started. It will listen on this port. If K8S_NODEJS_PORT is defined then that takes precedence |
| METRICS_HTTP_PORT | 5055 | false | prometheus metrics for the proxy app are exposed on this port. It should be different than http/https port on which application listens on |
| CERT_KEY_FILE | false | private key file | |
| CERT_FILE | false | public key file | |
| CERT_CA_FILE | false | CA certificate file | |
| CERT_KEY_PASSWD_FILE | false | password for private key is read from this file | |
| APP_URL_PREFIX | false | prefix context path. if set all URIs will be prefixed with this context. e.g. if value is set to v1 then URIs will be /v1/mproxy/ and /v1/kubesd/ |
Note: when metrics are exposed over https and certificates does not have pod ip as Subject name or Subject Alternative Name then the call to get metrics will fail giving invalid certificate error. In this case you can set environment variable NODE_TLS_REJECT_UNAUTHORIZED=0 which will disable certificate validation.
We welcome Your interest in the American Express Open Source Community on Github. Any Contributor to any Open Source Project managed by the American Express Open Source Community must accept and sign an Agreement indicating agreement to the terms below. Except for the rights granted in this Agreement to American Express and to recipients of software distributed by American Express, You reserve all right, title, and interest, if any, in and to Your Contributions. Please fill out the Agreement.
Any contributions made under this project will be governed by the Apache License 2.0.
This project adheres to the American Express Community Guidelines. By participating, you are expected to honor these guidelines.