Skip to content

Commit 98daa43

Browse files
awgreeneawgreene
committed
Introduce PPROF Documentation
This commit introduces documentation detailing how to retrieve PPROF data from the OLM and Catalog operators.
1 parent 99f7b5e commit 98daa43

File tree

1 file changed

+76
-40
lines changed

1 file changed

+76
-40
lines changed

‎content/en/docs/Tasks/performance-profiling.md‎

Lines changed: 76 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
---
2-
title: "Continuous Profiling"
2+
title: "Performance Profiling Metrics"
33
weight: 10
44
description: >
5-
The goal of this document is to familiarize you with OLM's stance on performance profiling.
5+
The goal of this document is to familiarize you with the steps to enable and review OLM's performance profiling metrics.
66
---
77

88
## Prerequisites
@@ -11,60 +11,96 @@ description: >
1111

1212
## Background
1313

14-
OLM utilizes the [pprof package](https://golang.org/pkg/net/http/pprof/) from the standard go library to expose performance profiles for the OLM Operator, the Catalog Operator, and Registry Servers. Due to the sensitive nature of this data, client requests against the pprof endpoint are rejected unless they are made with the certificate data kept in the `pprof-cert secret` in the `olm namespace`.
15-
Kubernetes does not provide a native way to prevent pods on cluster from iterating over the list of available ports and retrieving the data exposed. Without authetnicating the requests, OLM could leak customer usage statistics on multitenant clusters. If the aforementioned secret does not exist the pprof data will not be accessable.
14+
OLM utilizes the [pprof package](https://golang.org/pkg/net/http/pprof/) from the standard go library to expose performance profiles for the OLM Operator, the Catalog Operator, and Registry Servers. Due to the sensitive nature of this data, OLM must be configured to use TLS Certificates before performance profiling can be enabled.
1615

17-
### Retrieving PProf Data
16+
Requests against the performance profiling endpoint will be rejected unless the client certificate is validated by OLM. Unfortunately, Kubernetes does not provide a native way to prevent pods on cluster from iterating over the list of available ports and retrieving the data exposed. Without authenticating the requests, OLM could leak customer usage statistics on multitenant clusters.
1817

19-
#### OLM Operator
18+
This document will dive into the steps to [enable olm performance profiling](enable-performance-profiling) and retrieving pprof data from each component.
2019

21-
```bash
22-
$ go tool pprof http://localhost:8080/debug/pprof/heap #TODO: Replace with actual command
23-
```
20+
## Enabling Performance Profiling
21+
22+
### Creating a Certificate
2423

25-
#### Catalog Operator
24+
A valid server certiciate must be created for each component before Performance Profiling can be enabled. If you are unfamiliar with certificate generation, I recomend using the [OpenSSL](https://www.openssl.org/) tool-kit and refer to the [request certificate](https://www.openssl.org/docs/manmaster/man1/openssl-req.html) documentation.
25+
26+
Once you have generated a private and public key, this data should be stored in a `TLS Secret`:
2627

2728
```bash
28-
$ go tool pprof http://localhost:8080/debug/pprof/heap #TODO: Replace with actual command
29+
30+
$ export TLS_SECRET=my-tls-secret
31+
$ export PRIVATE_KEY_FILENAME=private.key # Replace with the name of the file that contains the private key you generated.
32+
$ export PUBLIC_KEY_FILENAME=certificate.crt # Replace with the name of the file that contains the public key you generated.
33+
34+
$ cat << EOF | kubectl apply -f -
35+
apiVersion: v1
36+
kind: Secret
37+
metadata:
38+
name: $TLS_SECRET
39+
namespace: olm
40+
type: kubernetes.io/tls
41+
data:
42+
tls.key: $(base64 -w0 $PRIVATE_KEY_FILENAME)
43+
tls.crt: $(base64 -w0 $PUBLIC_KEY_FILENAME)
44+
EOF
2945
```
3046

31-
#### Registry Server
47+
### Updating OLM to Use the TLS Secret
48+
49+
Patch the OLM Deployment's pod template to use the generated TLS secret:
50+
51+
- Defining a volume and volumeMount
52+
- Adding the `client-ca`, `tls-key` and `tls-crt` arguments
53+
- Replacing all mentions of port `8080` with `8443`
54+
- Updating the `livenessProbe` and `readinessProbe` to use HTTPS as the scheme.
55+
56+
This can by generating patching an existing OLM deployment:
3257

3358
```bash
34-
$ go tool pprof http://localhost:8080/debug/pprof/heap #TODO: Replace with actual command
35-
```
59+
$ export TLS_SECRET=my-tls-secret
60+
$ export CERT_PATH=/var/run/secrets # Define where to mount the certs.
61+
# Set Deployment name to olm-operator or catalog-operator
62+
$ export DEPLOYMENT_NAME=olm-operator
63+
64+
$ kubectl patch deployment $DEPLOYMENT_NAME -n olm --type json -p='[
65+
# Mount the secret to the pod
66+
{"op": "add", "path": "/spec/template/spec/volumes", "value":[{"name": '$TLS_SECRET', "secret": {"secretName": '$TLS_SECRET'}}]},
67+
{"op": "add", "path": "/spec/template/spec/containers/0/volumeMounts", "value":[{"name": '$TLS_SECRET', "mountPath": '$CERT_PATH'}]},
68+
69+
# Add startup arguments
70+
{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"--client-ca"},
71+
{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"'$CERT_PATH'/tls.crt"},
72+
{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"--tls-key"},
73+
{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"'$CERT_PATH'/tls.key"},
74+
{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"--tls-cert"},
75+
{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"'$CERT_PATH'/tls.crt"},
76+
77+
# Replace port 8080 with 8443
78+
{"op": "replace", "path": "/spec/template/spec/containers/0/ports/0", "value":{"containerPort": 8443}},
79+
{"op": "replace", "path": "/spec/template/spec/containers/0/livenessProbe/httpGet/port", "value":8443},
80+
{"op": "replace", "path": "/spec/template/spec/containers/0/readinessProbe/httpGet/port", "value":8443},
81+
82+
# Update livenessProbe and readinessProbe to use HTTPS
83+
{"op": "replace", "path": "/spec/template/spec/containers/0/readinessProbe/httpGet/scheme", "value":"HTTPS"},
84+
{"op": "replace", "path": "/spec/template/spec/containers/0/livenessProbe/httpGet/scheme", "value":"HTTPS"},
85+
]'
86+
deployment.apps/olm-operator patched
3687

37-
<details>
38-
<summary>Downstream docs, click to expand!</summary>
39-
40-
## Continuous Profiling
41-
OLM relies on [pprof-dump]() to periodically collect the pprof data and store it in the contents of a `ConfigMap`. The data in these `ConfigMaps` may be referenced when debugging issues.
88+
```
4289

43-
### Default PProf-Dump Settings
90+
## Accessing PPROF Data
4491

45-
OLM configures pprof-dump with the `pprof-dump ConfigMap` setting the follow default configurations:
92+
You will need to be able to access OLM port, for dev purposes the following commands may prove useful:
4693

47-
```yaml
48-
kind: ConfigMap
49-
metadata:
50-
name: prof-dump
51-
namespace: olm
52-
Data:
53-
garbageCollection: 60 # Delete configmaps older than 60 minutes
54-
poll: 15 # interval in minutes that pprof data is collected and dumped into ConfigMaps
94+
```bash
95+
# Set Deployment name to olm-operator or catalog-operator
96+
$ export DEPLOYMENT_NAME=olm-operator
97+
$ kubectl port-forward deployment/$DEPLOYMENT_NAME 8443:8443 -n olm
5598
```
5699

57-
### How do I disable continuous profiling?
100+
You can then curl the OLM `/debug/pprof` endpoint to retrieve default pprof profiles like so:
58101

59-
To disable OLM's continuous profiling, apply the following YAML:
102+
```bash
103+
$ curl https://localhost:8443/debug/pprof/heap --cert certificate.crt --key private.key --insecure -o olm-heap
60104

61-
```yaml
62-
kind: ConfigMap
63-
metadata:
64-
name: prof-dump
65-
namespace: olm
66-
Data:
67-
garbageCollection: 60
68-
poll: 0
105+
$ go tool pprof --top olm-heap
69106
```
70-
</details>

0 commit comments

Comments
 (0)