Skip to content
This repository has been archived by the owner on Feb 22, 2022. It is now read-only.

[incubator/kafka] Makes persistence fully configurable #3721

Merged
merged 11 commits into from
Feb 28, 2018
Merged

[incubator/kafka] Makes persistence fully configurable #3721

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
680934e
Makes kafka persistentVolume configurable
lsthornt Feb 13, 2018
8ee0300
Downgrades version to 0.3.0
lsthornt Feb 13, 2018
e790c35
Cleans up
fedenusy Feb 13, 2018
3d4a4b4
Fixes indentation
fedenusy Feb 13, 2018
1dba9a6
Adds missing spaces for helm lint
fedenusy Feb 13, 2018
581b8ee
Merge branch 'master' into kafka-toggle-persistence
fedenusy Feb 22, 2018
a819a79
Goes back to volumeClaimTemplates to support scaling
fedenusy Feb 22, 2018
00a2901
Merge remote-tracking branch 'origin/kafka-toggle-persistence' into k…
fedenusy Feb 22, 2018
218de82
Adds missing whitespace eater
fedenusy Feb 22, 2018
cf49f93
Follows best practices for defining storageClassName
fedenusy Feb 26, 2018
99a9d53
Merge branch 'master' into kafka-toggle-persistence
lsthornt Feb 28, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion incubator/kafka/Chart.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ apiVersion: v1
description: Apache Kafka is publish-subscribe messaging rethought as a distributed
commit log.
name: kafka
version: 0.2.13
version: 0.3.0
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bumped minor version because some of the top-level config values nested underneath persistence. Otherwise, could be 0.2.14

keywords:
- kafka
- zookeeper
Expand Down
42 changes: 24 additions & 18 deletions incubator/kafka/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,24 +51,28 @@ This chart includes a ZooKeeper chart as a dependency to the Kafka
cluster in its `requirement.yaml` by default. The chart can be customized using the
following configurable parameters:

| Parameter | Description | Default |
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly whitespace changes here, because the #brokerconfigs link got extracted

| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------- |
| `image` | Kafka Container image name | `solsson/kafka` |
| `imageTag` | Kafka Container image tag | `1.0.0` |
| `imagePullPolicy` | Kafka Container pull policy | `Always` |
| `replicas` | Kafka Brokers | `3` |
| `component` | Kafka k8s selector key | `kafka` |
| `resources` | Kafka resource requests and limits | `{}` |
| `dataDirectory` | Kafka data directory | `/opt/kafka/data` |
| `logSubPath` | Subpath under `dataDirectory` where kafka logs will be placed. `logs/` | `logs` |
| `affinity` | Pod scheduling preferences | `{}` |
| `storage` | Kafka Persistent volume size | `1Gi` |
| `configurationOverrides` | `Kafka ` [configuration setting](https://kafka.apache.org/documentation/#brokerconfigs) overrides in the dictionary format `setting.name: value` | `{}` |
| `schema-registry.enabled` | If True, installs Schema Registry Chart | `false` |
| `updateStrategy` | StatefulSet update strategy to use. | `{ type: "OnDelete" }` |
| `zookeeper.enabled` | If True, installs Zookeeper Chart | `true` |
| `zookeeper.url` | URL of Zookeeper Cluster (unneeded if installing Zookeeper Chart) | `""` |
| `zookeeper.port` | Port of Zookeeper Cluster | `2181` |
| Parameter | Description | Default |
| ------------------------------ | --------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- |
| `image` | Kafka Container image name | `solsson/kafka` |
| `imageTag` | Kafka Container image tag | `1.0.0` |
| `imagePullPolicy` | Kafka Container pull policy | `Always` |
| `replicas` | Kafka Brokers | `3` |
| `component` | Kafka k8s selector key | `kafka` |
| `resources` | Kafka resource requests and limits | `{}` |
| `logSubPath` | Subpath under `persistence.mountPath` where kafka logs will be placed. | `logs` |
| `affinity` | Pod scheduling preferences | `{}` |
| `configurationOverrides` | `Kafka ` [configuration setting][brokerconfigs] overrides in the dictionary format | `setting.name: value` | `{}` |
| `persistence.enabled` | Use a PVC to persist data | `true` |
| `persistence.existingClaim` | Provide an existing PersistentVolumeClaim | `nil` |
| `persistence.size` | Size of data volume | `1Gi` |
| `persistence.mountPath` | Mount path of data volume | `/opt/kafka/data` |
| `persistence.storageClass` | Storage class of backing PVC | `nil` |
| `persistence.annotations` | Persistent Volume annotations | `{}` |
| `schema-registry.enabled` | If True, installs Schema Registry Chart | `false` |
| `updateStrategy` | StatefulSet update strategy to use. | `{ type: "OnDelete" }` |
| `zookeeper.enabled` | If True, installs Zookeeper Chart | `true` |
| `zookeeper.url` | URL of Zookeeper Cluster (unneeded if installing Zookeeper Chart) | `""` |
| `zookeeper.port` | Port of Zookeeper Cluster | `2181` |

Specify parameters using `--set key=value[,key=value]` argument to `helm install`

Expand Down Expand Up @@ -111,3 +115,5 @@ Where `my-release` is the name of your helm release.
* Topic creation is not automated
* Only supports storage options that have backends for persistent volume claims (tested mostly on AWS)
* Kafka cluster is not accessible via an external endpoint

[brokerconfigs]: https://kafka.apache.org/documentation/#brokerconfigs
2 changes: 1 addition & 1 deletion incubator/kafka/templates/configmap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ metadata:
chart: "{{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}"
data:
zookeeper.connect: "{{ template "zookeeper.url" . }}"
log.dirs: "{{ printf "%s/%s" .Values.dataDirectory .Values.logSubPath }}"
log.dirs: "{{ printf "%s/%s" .Values.persistence.mountPath .Values.logSubPath }}"
{{- range $configName, $configValue := .Values.configurationOverrides }}
{{ $configName }}: "{{ $configValue -}}"
{{- end -}}
28 changes: 28 additions & 0 deletions incubator/kafka/templates/pvc.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
{{- if and .Values.persistence.enabled (not .Values.persistence.existingClaim) -}}
kind: "PersistentVolumeClaim"
apiVersion: "v1"
metadata:
name: {{ include "kafka.fullname" . | quote }}
labels:
app: {{ include "kafka.fullname" . | quote }}
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
release: "{{ .Release.Name }}"
heritage: "{{ .Release.Service }}"
{{- if .Values.persistence.annotations }}
annotations:
{{ toYaml .Values.persistence.annotations | indent 4 }}
{{- end }}
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: {{ .Values.persistence.size | quote }}
{{- if .Values.persistence.storageClass }}
{{- if (eq "-" .Values.persistence.storageClass) }}
storageClassName: ""
{{- else }}
storageClassName: {{ .Values.persistence.storageClass | quote }}
{{- end }}
{{- end }}
{{- end -}}
21 changes: 9 additions & 12 deletions incubator/kafka/templates/statefulset.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,15 +82,12 @@ spec:
/etc/confluent/docker/run
volumeMounts:
- name: datadir
mountPath: "{{ .Values.dataDirectory }}"
volumeClaimTemplates:
- metadata:
name: datadir
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: {{ .Values.storage }}
{{- if .Values.storageClass }}
storageClassName: {{ .Values.storageClass | quote }}
{{- end }}
mountPath: {{ .Values.persistence.mountPath | quote }}
volumes:
- name: datadir
{{- if .Values.persistence.enabled }}
persistentVolumeClaim:
claimName: {{ .Values.persistence.existingClaim | default (include "kafka.fullname" .) }}
{{- else }}
emptyDir: {}
{{- end }}
45 changes: 35 additions & 10 deletions incubator/kafka/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,6 @@ resources: {}
# requests:
# cpu: 100m
# memory: 1024Mi
#
## The size of the persistentVolume to allocate to each Kafka Pod in the StatefulSet. For
## production servers this number should likely be much larger.
storage: "1Gi"

## The StatefulSet Update Strategy which Kafka will use when changes are applied.
## ref: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#update-strategies
Expand All @@ -37,12 +33,8 @@ updateStrategy:
## The name of the storage class which the cluster should use.
# storageClass: default

## The location within the Kafka container where the PV will mount its storage and Kafka will store
## its logs
dataDirectory: "/opt/kafka/data"

## The subpath within the Kafka container's PV where logs will be stored
## This is combined with `dataDirectory` above, to create, by default: /opt/kafka/data/logs
## The subpath within the Kafka container's PV where logs will be stored.
## This is combined with `persistence.mountPath`, to create, by default: /opt/kafka/data/logs
logSubPath: "logs"

## Pod scheduling preferences.
Expand All @@ -57,6 +49,39 @@ affinity: {}
configurationOverrides:
"offsets.topic.replication.factor": 3

## Persistence configuration. Specify if and how to persist data to a persistent volume.
##
persistence:
enabled: true

## A manually managed Persistent Volume and Claim. Requires persistence.enabled: true.
## If defined, PVC must be created manually before volume will be bound
##
# existingClaim: "my-pvc-name"

## The size of the PersistentVolume to allocate to each Kafka Pod in the StatefulSet. For
## production servers this number should likely be much larger.
##
size: "1Gi"

## The location within the Kafka container where the PV will mount its storage and Kafka will
## store its logs.
##
mountPath: "/opt/kafka/data"

## The storageClass to use for Kafka's PersistentVolume
## - If defined, storageClassName: <storageClass>
## - If set to "-", storageClassName: "", which disables dynamic provisioning
## - If undefined (the default) or set to null, no storageClassName spec is
## set, choosing the default provisioner. (gp2 on AWS, standard on
## GKE, AWS & OpenStack)
##
# storageClass: "-"

## Custom PVC annotations.
##
# annotations: {}

# ------------------------------------------------------------------------------
# Zookeeper:
# ------------------------------------------------------------------------------
Expand Down