Skip to content

Commit bdcbb7f

Browse files
marcel-dempersmarcel-dempers
committed
add thanos to prometheus and query service
1 parent 221de9f commit bdcbb7f

File tree

5 files changed

+176
-40
lines changed

5 files changed

+176
-40
lines changed

‎monitoring/prometheus/kubernetes/thanos/README.md‎

Lines changed: 73 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ To follow this walkthrough , you will need [Kubernetes Monitoring](../1.33/READM
88
You may need to build the applications first using docker
99

1010
```
11-
docker compose -f monitoring\prometheus\docker-compose.yaml build
11+
docker compose -f monitoring/prometheus/docker-compose.yaml build
1212
```
1313

1414
Load the images into our cluster created in the monitoring guide:
@@ -18,6 +18,7 @@ kind load docker-image docker.io/library/go-application:latest --name monitoring
1818
kind load docker-image docker.io/library/dotnet-application:latest --name monitoring
1919
kind load docker-image docker.io/library/python-application:latest --name monitoring
2020
kind load docker-image docker.io/library/nodejs-application:latest --name monitoring
21+
2122
```
2223

2324
Deploy our microservices:
@@ -27,74 +28,110 @@ kubectl apply -f monitoring/prometheus/go-application/deployment.yaml
2728
kubectl apply -f monitoring/prometheus/dotnet-application/deployment.yaml
2829
kubectl apply -f monitoring/prometheus/python-application/deployment.yaml
2930
kubectl apply -f monitoring/prometheus/nodejs-application/deployment.yaml
30-
```
3131
32-
## Deploy Prometheus Instances:
32+
# see all our apps
33+
kubectl get pods
34+
NAME READY STATUS RESTARTS AGE
35+
dotnet-application-74dbc8b5d9-tjtxk 1/1 Running 0 3s
36+
go-application-65bbc698f-92jdr 1/1 Running 0 2m7s
37+
nodejs-application-c47c5f4c8-b9jhg 1/1 Running 0 2m7s
38+
python-application-759b44fff7-5fn8r 1/1 Running 0 2m7s
39+
```
3340

41+
## Create Service Monitors
3442

35-
We will need a service account with RBAC permissions for our new Prometheus instances to access service monitors and allow scraping of service endpoints
36-
In this guide I will use a single service account in two Prometheus instance:
43+
We'll need `ServiceMonitor` objects to point to our applications. In my guide I generally have one `ServiceMonitor` for each application.
3744

3845
```
39-
kubectl apply -f monitoring/prometheus/kubernetes/prometheus-operator/serviceaccount.yaml
46+
kubectl apply -f monitoring/prometheus/kubernetes/prometheus-operator/servicemonitors.yaml
4047
```
4148

42-
To showcase Thanos, we want to have more than one instance of `Prometheus` running, so that we can demonstrate the sidecar functionality & the global query view that Thanos provides. </br>
49+
Each service monitor has an extra label which called `prometheus-shard` which indicates which Prometheus instance should scrape it. The manual shards will use this label to manually select services to scrape.
50+
Two service monitors will be scraped by instance `prometheus-00` and the other two will be scraped by `prometheus-01`.
4351

44-
Apply Prometheus instances. </br>
45-
We can either apply both manual shards, or use a single defined automated shard which will split into two `StatefulSet` objects. In this guide I will keep it simple and use two `Prometheus` instances.
52+
For automated sharding, the automated shard will select `ServiceMonitor`'s based on label `monitoring` and `Prometheus` will automatically split each of the selected `ServiceMonitor` items into different `StatefulSets` . </br>
53+
See my Prometheus Sharding video in Kubernetes for more on that. </br>
54+
55+
## Create an S3 storage for Thanos
56+
57+
Thanos needs an S3 compatible storage for storing its data and long term retention.
58+
59+
In this guide I have created a very basic simple S3 storage using [Minio](https://github.com/minio/minio). Please note my example is not a highly available minio instance and not production ready either.
60+
61+
Deploy our test Minio instance:
4662

4763
```
48-
kubectl apply -f monitoring/prometheus/kubernetes/thanos/prometheus-00.yaml
49-
kubectl apply -f monitoring/prometheus/kubernetes/thanos/prometheus-00.yaml
64+
kubectl apply -f monitoring/prometheus/kubernetes/thanos/minio.yaml
65+
```
66+
67+
This will create an S3 storage for testing purpose and a `Job` that will create a bucket for our Thanos data. </br>
68+
69+
## Create Thanos storage secret
70+
71+
Before we apply Prometheus instances, we will need a secret that Thanos sidecars in each Prometheus instance will use to connect to store data in S3.
5072

73+
Let's create a secret for minio:
74+
75+
```
76+
kubectl apply -f monitoring/prometheus/kubernetes/thanos/thanos-secret.yaml
5177
```
5278

53-
Apply Service Monitors
79+
## Deploy Prometheus Instances:
80+
81+
We will need a service account with RBAC permissions for our new Prometheus instances to access service monitors and allow scraping of service endpoints
82+
In this guide I will use a single service account in two Prometheus instance:
5483

5584
```
56-
kubectl apply -f monitoring/prometheus/kubernetes/prometheus-operator/servicemonitors.yaml
85+
kubectl apply -f monitoring/prometheus/kubernetes/prometheus-operator/serviceaccount.yaml
5786
```
5887

59-
Each service monitor has an extra label which called `prometheus-shard` which indicates which Prometheus instance should scrape it. The manual shards will use this label to manually select services to scrape.
60-
Two service monitors will be scraped by instance `prometheus-00` and the other two will be scraped by `prometheus-01`.
88+
To showcase Thanos, we want to have more than one instance of `Prometheus` running, so that we can demonstrate the sidecar functionality & the global query view that Thanos provides. </br>
6189

62-
For automated sharding, the automated shard will select `ServiceMonitor`'s based on label `monitoring` and `Prometheus` will automatically split each of the selected `ServiceMonitor` items into different `StatefulSets` . </br>
90+
We can either apply both manual shards, or use a single defined automated shard which will split into two `StatefulSet` objects. In this guide I will keep it simple and use two `Prometheus` instances.
6391

64-
We can now see our Prometheus instances in the `default` namespace:
92+
To understand Thanos, we'll take a closer look at the `Prometheus` spec in our instances we are going to deploy:
6593

6694
```
67-
NAME READY STATUS RESTARTS AGE
68-
dotnet-application-74dbc8b5d9-fq2p5 1/1 Running 0 24h
69-
go-application-65bbc698f-jxnls 1/1 Running 0 24h
70-
nodejs-application-c47c5f4c8-cj7jl 1/1 Running 0 24h
71-
prometheus-prometheus-0 2/2 Running 0 17h
72-
prometheus-prometheus-00-0 2/2 Running 0 23h
73-
prometheus-prometheus-01-0 2/2 Running 0 23h
74-
prometheus-prometheus-shard-1-0 2/2 Running 0 17h
75-
python-application-759b44fff7-s276f 1/1 Running 0 24h
95+
kubectl apply -f monitoring/prometheus/kubernetes/thanos/prometheus-00.yaml
96+
kubectl apply -f monitoring/prometheus/kubernetes/thanos/prometheus-01.yaml
97+
7698
```
7799

78-
Checkout each of the automated Prometheus shards
100+
We can now see our Prometheus instances in the `default` namespace:
79101

80102
```
81-
kubectl port-forward prometheus-prometheus-0 9090
82-
kubectl port-forward prometheus-prometheus-shard-1-0 9091:9090
103+
kubectl get pods
104+
NAME READY STATUS RESTARTS AGE
105+
dotnet-application-74dbc8b5d9-tjtxk 1/1 Running 0 60m
106+
go-application-65bbc698f-92jdr 1/1 Running 0 62m
107+
nodejs-application-c47c5f4c8-b9jhg 1/1 Running 0 62m
108+
prometheus-prometheus-00-0 3/3 Running 0 21s
109+
prometheus-prometheus-01-0 3/3 Running 0 17s
110+
python-application-759b44fff7-5fn8r 1/1 Running 0 62m
83111
```
84112

85-
Now that we have applications to monitor and we have `Prometheus` instances scraping these applications in different instances, we can finally implement Thanos into our solution
113+
Checkout each of the `Prometheus` instances and see our 4 applications sharded to 2 instances evenly. </br>
86114

115+
```
116+
kubectl port-forward prometheus-prometheus-00-0 9090
117+
kubectl port-forward prometheus-prometheus-01-0 9091:9090
118+
```
87119

88-
## Create an S3 storage for Thanos
120+
## Deploying the Thanos Query Service
89121

90-
Thanos needs an S3 compatible storage for storing its data and long term retention.
122+
Thanos is now enabled on our `Prometheus` instances, which means the sidecars are shipping metrics data from each instance to our S3. </br>
123+
For us to create a global query service we need to deploy the Thanos Query component which will link up to each Thanos sidecar, using the service created by the Prometheus operator
91124

92-
In this guide I have created a very basic simple S3 storage using [Minio](https://github.com/minio/minio). Please note my example is not a highly available minio instance and not production ready either.
125+
```
126+
kubectl apply -f monitoring/prometheus/kubernetes/thanos/thanos-query.yaml
127+
```
93128

94-
Deploy our test Minio instance:
129+
## Test in Grafana
130+
131+
We can now verify that we can query Thanos in Grafana by using `port-forward` to access Grafana.
95132

96133
```
97-
kubectl apply -f monitoring/prometheus/kubernetes/thanos/minio.yaml
134+
kubectl -n monitoring port-forward svc/grafana 3000:80
98135
```
99136

100-
This will create an S3 storage for testing purpose and a `Job` that will create a bucket for our Thanos data. </br>
137+
Then access Grafana on [localhost:3000](http://localhost:3000/)

‎monitoring/prometheus/kubernetes/thanos/prometheus-00.yaml‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ spec:
1111
nodeSelector:
1212
kubernetes.io/os: linux
1313
replicas: 1
14-
retention: 30d
14+
retention: 2d # Thanos will now have long term storage
1515
scrapeInterval: 30s
1616
serviceAccountName: prometheus-applications
1717
serviceMonitorNamespaceSelector:
@@ -23,4 +23,15 @@ spec:
2323
shards: 1
2424
version: v3.4.1
2525
externalLabels:
26-
prometheus-shard: prometheus-00
26+
prometheus: applications
27+
prometheus-shard: prometheus-00
28+
thanos:
29+
baseImage: quay.io/thanos/thanos:v0.38.0
30+
objectStorageConfig:
31+
name: thanos-secret
32+
key: objstore.yml
33+
securityContext:
34+
fsGroup: 2000
35+
runAsGroup: 2000
36+
runAsNonRoot: true
37+
runAsUser: 1000

‎monitoring/prometheus/kubernetes/thanos/prometheus-01.yaml‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ spec:
1111
nodeSelector:
1212
kubernetes.io/os: linux
1313
replicas: 1
14-
retention: 30d
14+
retention: 2d # Thanos will now have long term storage
1515
scrapeInterval: 30s
1616
serviceAccountName: prometheus-applications
1717
serviceMonitorNamespaceSelector:
@@ -23,4 +23,15 @@ spec:
2323
shards: 1
2424
version: v3.4.1
2525
externalLabels:
26-
prometheus-shard: prometheus-01
26+
prometheus: applications
27+
prometheus-shard: prometheus-01
28+
thanos:
29+
baseImage: quay.io/thanos/thanos:v0.38.0
30+
objectStorageConfig:
31+
name: thanos-secret
32+
key: objstore.yml
33+
securityContext:
34+
fsGroup: 2000
35+
runAsGroup: 2000
36+
runAsNonRoot: true
37+
runAsUser: 1000

‎monitoring/prometheus/kubernetes/thanos/thanos-query.yaml‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
name: thanos-query
5+
labels:
6+
app: thanos-query
7+
spec:
8+
replicas: 1
9+
selector:
10+
matchLabels:
11+
app: thanos-query
12+
template:
13+
metadata:
14+
labels:
15+
app: thanos-query
16+
spec:
17+
containers:
18+
- name: thanos-query
19+
image: quay.io/thanos/thanos:v0.38.0
20+
args:
21+
- "query"
22+
- "--http-address=0.0.0.0:19090" # Thanos Query HTTP API, for Grafana and UI
23+
- "--grpc-address=0.0.0.0:19091" # Thanos Query gRPC API, for other Thanos components
24+
# Connect to Prometheus Thanos sidecars.
25+
# These should be the Kubernetes Service names exposing the Thanos sidecar's gRPC port (10901 or 19091).
26+
- --endpoint=dnssrv+_grpc._tcp.prometheus-operated.default.svc.cluster.local
27+
ports:
28+
- name: http
29+
containerPort: 19090
30+
protocol: TCP
31+
- name: grpc
32+
containerPort: 19091
33+
protocol: TCP
34+
resources:
35+
requests:
36+
cpu: 100m
37+
memory: 256Mi
38+
limits:
39+
cpu: 500m
40+
memory: 1Gi
41+
# Optional: Add a service account if you need to grant specific RBAC permissions
42+
# serviceAccountName: thanos-query-sa
43+
---
44+
apiVersion: v1
45+
kind: Service
46+
metadata:
47+
name: thanos-query
48+
labels:
49+
app: thanos-query
50+
spec:
51+
selector:
52+
app: thanos-query
53+
ports:
54+
- name: http
55+
protocol: TCP
56+
port: 19090
57+
targetPort: http
58+
- name: grpc
59+
protocol: TCP
60+
port: 19091
61+
targetPort: grpc
62+
type: ClusterIP # Use ClusterIP for internal access. For external, consider NodePort or Ingress.

‎monitoring/prometheus/kubernetes/thanos/thanos-secret.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
apiVersion: v1
2+
kind: Secret
3+
metadata:
4+
name: thanos-secret
5+
type: Opaque
6+
stringData:
7+
objstore.yml: |
8+
type: s3
9+
config:
10+
bucket: "thanos-blocks"
11+
endpoint: "minio-service:9000"
12+
region: "us-east-1" # MinIO doesn't care about region, but S3 requires it
13+
access_key: "thanos"
14+
secret_key: "supersecretpassword"
15+
insecure: true # Use insecure connection for MinIO (HTTP, not HTTPS)

0 commit comments

Comments
 (0)