Skip to content

Commit 7e9b30a

Browse files
awgreeneawgreene
committed
wip
1 parent 99f7b5e commit 7e9b30a

File tree

1 file changed

+69
-41
lines changed

1 file changed

+69
-41
lines changed

‎content/en/docs/Tasks/performance-profiling.md‎

Lines changed: 69 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
---
2-
title: "Continuous Profiling"
2+
title: "Performance Profiling Metrics"
33
weight: 10
44
description: >
5-
The goal of this document is to familiarize you with OLM's stance on performance profiling.
5+
The goal of this document is to familiarize you with the steps to enable and review OLM's performance profiling metrics.
66
---
77

88
## Prerequisites
@@ -11,60 +11,88 @@ description: >
1111

1212
## Background
1313

14-
OLM utilizes the [pprof package](https://golang.org/pkg/net/http/pprof/) from the standard go library to expose performance profiles for the OLM Operator, the Catalog Operator, and Registry Servers. Due to the sensitive nature of this data, client requests against the pprof endpoint are rejected unless they are made with the certificate data kept in the `pprof-cert secret` in the `olm namespace`.
15-
Kubernetes does not provide a native way to prevent pods on cluster from iterating over the list of available ports and retrieving the data exposed. Without authetnicating the requests, OLM could leak customer usage statistics on multitenant clusters. If the aforementioned secret does not exist the pprof data will not be accessable.
14+
OLM utilizes the [pprof package](https://golang.org/pkg/net/http/pprof/) from the standard go library to expose performance profiles for the OLM Operator, the Catalog Operator, and Registry Servers. Due to the sensitive nature of this data, OLM must be configured to use TLS Certificates before performance profiling can be enabled.
1615

17-
### Retrieving PProf Data
16+
Requests against the performance profiling endpoint will be rejected unless the client certificate is validated by OLM. Unfortunately, Kubernetes does not provide a native way to prevent pods on cluster from iterating over the list of available ports and retrieving the data exposed. Without authetnicating the requests, OLM could leak customer usage statistics on multitenant clusters.
1817

19-
#### OLM Operator
18+
This document will dive into the steps to [enable olm performance profiling](enable-performance-profiling) and retrieving pprof data from each component.
2019

21-
```bash
22-
$ go tool pprof http://localhost:8080/debug/pprof/heap #TODO: Replace with actual command
23-
```
20+
## Enabling Performance Profiling
2421

25-
#### Catalog Operator
22+
### Creating a Certificate
2623

27-
```bash
28-
$ go tool pprof http://localhost:8080/debug/pprof/heap #TODO: Replace with actual command
29-
```
24+
A valid server certiciate must be created for each component before Performance Profiling can be enabled. If you are unfamiliar with certificate generation, I recomend using the [OpenSSL](https://www.openssl.org/) tool-kit and refer to the [request certificate](https://www.openssl.org/docs/manmaster/man1/openssl-req.html) documentation.
3025

31-
#### Registry Server
26+
Once you have generated a private and public key, this data should be stored in a `TLS Secret`:
3227

3328
```bash
34-
$ go tool pprof http://localhost:8080/debug/pprof/heap #TODO: Replace with actual command
29+
$ export PRIVATE_KEY_FILENAME=private.key # Replace with the name of the file that contains the private key you generated.
30+
$ export PUBLIC_KEY_FILENAME=certificate.key # Replace with the name of the file that contains the public key you generated.
31+
32+
$ cat << EOF | kubectl apply -f -
33+
apiVersion: v1
34+
kind: Secret
35+
metadata:
36+
name: olm-serving-secret
37+
namespace: olm
38+
type: kubernetes.io/tls
39+
data:
40+
tls.key: $(base64 $PRIVATE_KEY_FILENAME)
41+
tls.crt: $(base64 $PUBLIC_KEY_FILENAME)
42+
EOF
3543
```
3644

37-
<details>
38-
<summary>Downstream docs, click to expand!</summary>
39-
40-
## Continuous Profiling
41-
OLM relies on [pprof-dump]() to periodically collect the pprof data and store it in the contents of a `ConfigMap`. The data in these `ConfigMaps` may be referenced when debugging issues.
45+
### Retrieving the Performance Profile from the OLM Deployment
4246

43-
### Default PProf-Dump Settings
47+
Patch the OLM Deployment's pod template to use the generated TLS secret:
4448

45-
OLM configures pprof-dump with the `pprof-dump ConfigMap` setting the follow default configurations:
49+
- Defining a volume and volumeMount
50+
- Adding the `client-ca`, `tls-key` and `tls-crt` arguments
51+
- Replacing all mentions of port `8080` with `8443`
52+
- Updating the `livenessProbe` and `readinessProbe` to use HTTPS as the scheme.
4653

47-
```yaml
48-
kind: ConfigMap
49-
metadata:
50-
name: prof-dump
51-
namespace: olm
52-
Data:
53-
garbageCollection: 60 # Delete configmaps older than 60 minutes
54-
poll: 15 # interval in minutes that pprof data is collected and dumped into ConfigMaps
54+
This can be done with the following commands:
55+
56+
```bash
57+
$ export CERT_PATH=/etc/olm-serving-certs # Define where to mount the certs.
58+
$ kubectl patch deployment olm-operator -n olm --type json -p='[
59+
# Mount the secret to the pod
60+
{"op": "add", "path": "/spec/template/spec/volumes", "value":[{"name": "olm-serving-cert", "secret": {"secretName": "olm-serving-cert"}}]},
61+
{"op": "add", "path": "/spec/template/spec/containers/0/volumeMounts", "value":[{"name": "olm-serving-cert", "mountPath": '$CERT_PATH'}]},
62+
63+
# Add startup arguments
64+
{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"--client-ca"},
65+
{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"'$CERT_PATH'/tls.crt"},
66+
{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"--tls-key"},
67+
{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"'$CERT_PATH'/tls.key"},
68+
{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"--tls-cert"},
69+
{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"'$CERT_PATH'/tls.crt"},
70+
71+
# Replace port 8080 with 8443
72+
{"op": "replace", "path": "/spec/template/spec/containers/0/ports/0", "value":{"containerPort": 8443}},
73+
{"op": "replace", "path": "/spec/template/spec/containers/0/livenessProbe/httpGet/port", "value":8443},
74+
{"op": "replace", "path": "/spec/template/spec/containers/0/readinessProbe/httpGet/port", "value":8443},
75+
76+
# Update livenessProbe and readinessProbe to use HTTPS
77+
{"op": "replace", "path": "/spec/template/spec/containers/0/readinessProbe/httpGet/scheme", "value":"HTTPS"},
78+
{"op": "replace", "path": "/spec/template/spec/containers/0/livenessProbe/httpGet/scheme", "value":"HTTPS"},
79+
]'
80+
deployment.apps/olm-operator patched
5581
```
5682

57-
### How do I disable continuous profiling?
83+
You will need to be able to access OLM port, for dev purposes the following commands may prove useful:
84+
85+
```bash
86+
$ kubectl port-forward deployment/olm-operator 8443:8443 -n olm
87+
```
5888

59-
To disable OLM's continuous profiling, apply the following YAML:
89+
You can then curl the OLM `/debug/pprof` endpoint to retrieve default pprof profiles like so:
6090

61-
```yaml
62-
kind: ConfigMap
63-
metadata:
64-
name: prof-dump
65-
namespace: olm
66-
Data:
67-
garbageCollection: 60
68-
poll: 0
91+
```bash
92+
export BASE_URL=https://localhost:8443/debug/pprof
93+
curl $BASE_URL/goroutine --cert certificate.crt --key private.key --insecure -o goroutine
94+
curl $BASE_URL/heap --cert certificate.crt --key private.key --insecure -o heap
95+
curl $BASE_URL/threadcreate --cert certificate.crt --key private.key --insecure -o threadcreate
96+
curl $BASE_URL/block --cert certificate.crt --key private.key --insecure -o block
97+
curl $BASE_URL/mutex --cert certificate.crt --key private.key --insecure -o mutex
6998
```
70-
</details>

0 commit comments

Comments
 (0)