Skip to content

Commit f4bf735

Browse files
awgreeneawgreene
committed
wip
1 parent 99f7b5e commit f4bf735

File tree

1 file changed

+67
-41
lines changed

1 file changed

+67
-41
lines changed

‎content/en/docs/Tasks/performance-profiling.md‎

Lines changed: 67 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
---
2-
title: "Continuous Profiling"
2+
title: "Performance Profiling Metrics"
33
weight: 10
44
description: >
5-
The goal of this document is to familiarize you with OLM's stance on performance profiling.
5+
The goal of this document is to familiarize you with the steps to enable and review OLM's performance profiling metrics.
66
---
77

88
## Prerequisites
@@ -11,60 +11,86 @@ description: >
1111

1212
## Background
1313

14-
OLM utilizes the [pprof package](https://golang.org/pkg/net/http/pprof/) from the standard go library to expose performance profiles for the OLM Operator, the Catalog Operator, and Registry Servers. Due to the sensitive nature of this data, client requests against the pprof endpoint are rejected unless they are made with the certificate data kept in the `pprof-cert secret` in the `olm namespace`.
15-
Kubernetes does not provide a native way to prevent pods on cluster from iterating over the list of available ports and retrieving the data exposed. Without authetnicating the requests, OLM could leak customer usage statistics on multitenant clusters. If the aforementioned secret does not exist the pprof data will not be accessable.
14+
OLM utilizes the [pprof package](https://golang.org/pkg/net/http/pprof/) from the standard go library to expose performance profiles for the OLM Operator, the Catalog Operator, and Registry Servers. Due to the sensitive nature of this data, OLM must be configured to use TLS Certificates before performance profiling can be enabled.
1615

17-
### Retrieving PProf Data
16+
Requests against the performance profiling endpoint will be rejected unless the client certificate is validated by OLM. Unfortunately, Kubernetes does not provide a native way to prevent pods on cluster from iterating over the list of available ports and retrieving the data exposed. Without authetnicating the requests, OLM could leak customer usage statistics on multitenant clusters.
1817

19-
#### OLM Operator
18+
This document will dive into the steps to [enable olm performance profiling](enable-performance-profiling) and retrieving pprof data from each component.
2019

21-
```bash
22-
$ go tool pprof http://localhost:8080/debug/pprof/heap #TODO: Replace with actual command
23-
```
20+
## Enabling Performance Profiling
2421

25-
#### Catalog Operator
22+
### Creating a certificate
2623

27-
```bash
28-
$ go tool pprof http://localhost:8080/debug/pprof/heap #TODO: Replace with actual command
29-
```
24+
A valid server certiciate must be created for each component before Performance Profiling can be enabled. If you are unfamiliar with certificate generation, I recomend using the [OpenSSL](https://www.openssl.org/) tool-kit and refer to the [request certificate](https://www.openssl.org/docs/manmaster/man1/openssl-req.html) documentation.
3025

31-
#### Registry Server
26+
Once you have generated a private and public key, this data should be stored in a `TLS Secret`:
3227

3328
```bash
34-
$ go tool pprof http://localhost:8080/debug/pprof/heap #TODO: Replace with actual command
29+
$ export PRIVATE_KEY_FILENAME=private.key # Replace with the name of the file that contains the private key you generated.
30+
$ export PUBLIC_KEY_FILENAME=certificate.key # Replace with the name of the file that contains the public key you generated.
31+
32+
$ cat << EOF | kubectl apply -f -
33+
apiVersion: v1
34+
kind: Secret
35+
metadata:
36+
name: olm-serving-secret
37+
namespace: olm
38+
type: kubernetes.io/tls
39+
data:
40+
tls.key: $(base64 $PRIVATE_KEY_FILENAME)
41+
tls.crt: $(base64 $PUBLIC_KEY_FILENAME)
42+
EOF
3543
```
3644

37-
<details>
38-
<summary>Downstream docs, click to expand!</summary>
39-
40-
## Continuous Profiling
41-
OLM relies on [pprof-dump]() to periodically collect the pprof data and store it in the contents of a `ConfigMap`. The data in these `ConfigMaps` may be referenced when debugging issues.
45+
We then need to patch the OLM Deployment to use the TLS secret that we generated by:
4246

43-
### Default PProf-Dump Settings
47+
- Adding a volume and volumeMount to the olm pod which mounts the secret.
48+
- Adding the `client-ca`, `tls-key` and `tls-crt` arguments to the OLM pod
49+
- Replacing all mentions of port `8080` with `8443`
50+
- Update the `livenessProbe` and `readinessProbe` to use HTTPS as the scheme.
4451

45-
OLM configures pprof-dump with the `pprof-dump ConfigMap` setting the follow default configurations:
52+
This can be done with the following commands:
4653

47-
```yaml
48-
kind: ConfigMap
49-
metadata:
50-
name: prof-dump
51-
namespace: olm
52-
Data:
53-
garbageCollection: 60 # Delete configmaps older than 60 minutes
54-
poll: 15 # interval in minutes that pprof data is collected and dumped into ConfigMaps
54+
```bash
55+
$ export CERT_PATH=/etc/olm-serving-certs # Define where to mount the certs.
56+
$ kubectl patch deployment olm-operator -n olm --type json -p='[
57+
# Mount the secret to the pod
58+
{"op": "add", "path": "/spec/template/spec/volumes", "value":[{"name": "olm-serving-cert", "secret": {"secretName": "olm-serving-cert"}}]},
59+
{"op": "add", "path": "/spec/template/spec/containers/0/volumeMounts", "value":[{"name": "olm-serving-cert", "mountPath": '$CERT_PATH'}]},
60+
61+
# Add startup arguments
62+
{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"--client-ca"},
63+
{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"'$CERT_PATH'/tls.crt"},
64+
{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"--tls-key"},
65+
{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"'$CERT_PATH'/tls.key"},
66+
{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"--tls-cert"},
67+
{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"'$CERT_PATH'/tls.crt"},
68+
69+
# Replace port 8080 with 8443
70+
{"op": "replace", "path": "/spec/template/spec/containers/0/ports/0", "value":{"containerPort": 8443}},
71+
{"op": "replace", "path": "/spec/template/spec/containers/0/livenessProbe/httpGet/port", "value":8443},
72+
{"op": "replace", "path": "/spec/template/spec/containers/0/readinessProbe/httpGet/port", "value":8443},
73+
74+
# Update livenessProbe and readinessProbe to use HTTPS
75+
{"op": "replace", "path": "/spec/template/spec/containers/0/readinessProbe/httpGet/scheme", "value":"HTTPS"},
76+
{"op": "replace", "path": "/spec/template/spec/containers/0/livenessProbe/httpGet/scheme", "value":"HTTPS"},
77+
]'
78+
deployment.apps/olm-operator patched
5579
```
5680

57-
### How do I disable continuous profiling?
81+
You will need to be able to access OLM port, for dev purposes the following commands may prove useful:
82+
83+
```bash
84+
$ kubectl port-forward deployment/olm-operator 8443:8443 -n olm
85+
```
5886

59-
To disable OLM's continuous profiling, apply the following YAML:
87+
You can then curl the OLM `/debug/pprof` endpoint to retrieve default pprof profiles like so:
6088

61-
```yaml
62-
kind: ConfigMap
63-
metadata:
64-
name: prof-dump
65-
namespace: olm
66-
Data:
67-
garbageCollection: 60
68-
poll: 0
89+
```bash
90+
export BASE_URL=https://localhost:8443/debug/pprof
91+
curl $BASE_URL/goroutine --cert certificate.crt --key private.key --insecure -o goroutine
92+
curl $BASE_URL/heap --cert certificate.crt --key private.key --insecure -o heap
93+
curl $BASE_URL/threadcreate --cert certificate.crt --key private.key --insecure -o threadcreate
94+
curl $BASE_URL/block --cert certificate.crt --key private.key --insecure -o block
95+
curl $BASE_URL/mutex --cert certificate.crt --key private.key --insecure -o mutex
6996
```
70-
</details>

0 commit comments

Comments
 (0)