This repository was archived by the owner on Feb 22, 2022. It is now read-only.
adds chart for kafka #144
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| # Patterns to ignore when building packages. | ||
| # This supports shell glob matching, relative path matching, and | ||
| # negation (prefixed with !). Only one pattern per line. | ||
| .DS_Store | ||
| # Common VCS dirs | ||
| .git/ | ||
| .gitignore | ||
| .bzr/ | ||
| .bzrignore | ||
| .hg/ | ||
| .hgignore | ||
| .svn/ | ||
| # Common backup files | ||
| *.swp | ||
| *.bak | ||
| *.tmp | ||
| *~ | ||
| # Various IDEs | ||
| .project | ||
| .idea/ | ||
| *.tmproj | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| apiVersion: v1 | ||
| description: A Kafka Helm chart for Kubernetes | ||
| name: kafka | ||
| version: 0.1.0 | ||
| keywords: | ||
| - kafka | ||
| - zookeeper | ||
| - kafka petset | ||
| home: https://kafka.apache.org/ | ||
| sources: | ||
| - https://github.com/kubernetes/charts/tree/master/incubator/zookeeper | ||
| - https://github.com/Yolean/kubernetes-kafka | ||
|
There was a problem hiding this comment. can you add |
||
| - https://github.com/solsson/dockerfiles/tree/master/kafka | ||
| maintainers: | ||
| - name: Faraaz Khan | ||
| email: faraaz@rationalizeit.us | ||
| icon: https://kafka.apache.org/images/logo.png | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,113 @@ | ||
| # Apache Kafka Helm Chart | ||
|
|
||
| This is an implementation of Kafka PetSet found here: | ||
|
|
||
| * https://github.com/Yolean/kubernetes-kafka | ||
|
|
||
| ## Pre Requisites: | ||
|
|
||
| * Kubernetes 1.3 with alpha APIs enabled and support for storage classes | ||
|
|
||
| * PV support on underlying infrastructure | ||
|
|
||
| * Requires at least `v2.0.0-beta.1` version of helm to support | ||
| dependency management with requirements.yaml | ||
|
|
||
| ## PetSet Details | ||
|
|
||
| * http://kubernetes.io/docs/user-guide/petset/ | ||
|
|
||
| ## PetSet Caveats | ||
|
|
||
| * http://kubernetes.io/docs/user-guide/petset/#alpha-limitations | ||
|
|
||
| ## Chart Details | ||
|
|
||
| This chart will do the following: | ||
|
|
||
| * Implement a dynamically scalable kafka cluster using Kubernetes | ||
| PetSets | ||
|
|
||
| * Implement a dynamically scalable zookeeper cluster as another Kubernetes PetSet required for the Kafka cluster above | ||
|
|
||
| ### Installing the Chart | ||
|
|
||
| To install the chart with the release name `my-release` in the default | ||
| namespace: | ||
|
|
||
| ``` | ||
| helm repo add incubator | ||
| http://storage.googleapis.com/kubernetes-charts-incubator | ||
| helm install --name my-kafka incubator/kafka | ||
| ``` | ||
|
|
||
| If using a dedicated namespace(recommended) then make sure the namespace | ||
| exists with: | ||
|
|
||
| ``` | ||
| kubectl create ns kafka | ||
| helm install --name my-kafka --set global.namespace=kafka incubator/kafka | ||
| ``` | ||
|
|
||
| This chart includes a ZooKeeper chart as a dependency to the Kafka | ||
| cluster in its `requirement.yaml`. The chart can be customized using the | ||
| following configurable parameters: | ||
|
|
||
| | Parameter | Description | Default | | ||
| | ----------------------- | ---------------------------------- | ---------------------------------------------------------- | | ||
| | `Name` | Kafka master name | `kf` | | ||
| | `Image` | Kafka Container image name | `solsson/kafka` | | ||
| | `ImageTag` | Kafka Container image tag | `0.10.0.1` | | ||
| | `ImagePullPolicy` | Kafka Container pull policy | `Always` | | ||
| | `Replicas` | Kafka Brokers | `3` | | ||
| | `Component` | Kafka k8s selector key | `kafka` | | ||
| | `Cpu` | Kafka container requested cpu | `100m` | | ||
| | `Memory` | Kafka container requested memory | `512Mi` | | ||
| | `MaxCpu` | Kafka container cpu limit | `200m` | | ||
| | `MaxMemory` | Kafka container memory limit | `1024Mi` | | ||
| | `DataDirectory` | Kafka data directory | `/opt/kafka/data` | | ||
| | `Storage` | Kafka Persistent volume size | `1Gi` | | ||
| | `zookeeper.Name` | Name of the Zookeeper Service | `zk` | | ||
|
|
||
| Specify parameters using `--set key=value[,key=value]` argument to `helm install` | ||
|
|
||
| Alternatively a YAML file that specifies the values for the parameters can be provided like this: | ||
|
|
||
| ```bash | ||
| $ helm install --name my-release -f values.yaml incubator/kafka | ||
| ``` | ||
|
|
||
| ### Connecting to Kafka | ||
|
|
||
| You can connect to Kafka by running a simple pod in the K8s cluster like this with a configuration like this: | ||
|
There was a problem hiding this comment. You may want to add a notes.txt file with these instructions. |
||
|
|
||
| ```yaml | ||
| apiVersion: v1 | ||
| kind: Pod | ||
| metadata: | ||
| name: testclient | ||
| namespace: kafka | ||
| spec: | ||
| containers: | ||
| - name: kafka | ||
| image: solsson/kafka:0.10.0.1 | ||
| command: | ||
| - sh | ||
| - -c | ||
| - "exec tail -f /dev/null" | ||
| ``` | ||
|
|
||
| Once you have the testclient pod above running, you can list all kafka | ||
| topics with: | ||
|
|
||
| ` kubectl exec testclient -- ./bin/kafka-topics.sh --zookeeper | ||
| my-release-zk:2181 --list` | ||
|
|
||
| Where `my-release` is the name of your helm release. | ||
|
|
||
| ## Known Limitations | ||
|
|
||
| * Namespace creation is not automated | ||
|
There was a problem hiding this comment. This is an easy fix. Will need to happen for stable. This is may be a reason why unit tests are failing. There was a problem hiding this comment. @faraazkhan any word on what these services are for? |
||
| * Topic creation is not automated | ||
| * Only supports storage options that have backends for persistent volume claims (tested mostly on AWS) | ||
| * Kafka cluster is not accessible via an external endpoint | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| You can connect to Kafka by running a simple pod in the K8s cluster like this with a configuration like this: | ||
|
There was a problem hiding this comment. Helm will specifically look for a while called |
||
|
|
||
| apiVersion: v1 | ||
| kind: Pod | ||
| metadata: | ||
| name: testclient | ||
| namespace: kafka | ||
|
There was a problem hiding this comment. I wouldnt set the namespace explicitly here. Users may not have configured this namespace for installation. You can make this a kubectl run command and avoid having the pod yaml here. Look at the postgresql chart as an example.
|
||
| spec: | ||
| containers: | ||
| - name: kafka | ||
| image: solsson/kafka:0.10.0.1 | ||
| command: | ||
| - sh | ||
| - -c | ||
| - "exec tail -f /dev/null" | ||
|
|
||
| Once you have the testclient pod above running, you can list all kafka | ||
| topics with: | ||
|
|
||
| kubectl -n kafka exec testclient -- ./bin/kafka-topics.sh --zookeeper my-release-zk:2181 --list | ||
|
|
||
| To create a new topic: | ||
| kubectl -n kafka exec testclient -- ./bin/kafka-topics.sh --zookeeper my-release-zookeeper-zk:2181 --topic test1 --create --partitions 1 --replication-factor 1 | ||
|
|
||
| To listen for messages on a topic: | ||
| kubectl -n kafka exec -ti testclient -- ./bin/kafka-console-consumer.sh --zookeeper my-release-zookeeper-zk:2181 --topic test1 --from-beginning | ||
|
|
||
| To stop the listener session above press: Ctrl+C | ||
|
|
||
| To start an interactive message producer session: | ||
| kubectl -n kafka exec -ti testclient -- ./bin/kafka-console-producer.sh --broker-list my-release-broker-kf.kafka.svc.cluster.local:9092 --topic test1 | ||
|
There was a problem hiding this comment. You can simplify the dns name there and make it |
||
|
|
||
| To create a message in the above session, simply type the message and press "enter" | ||
| To end the producer session try: Ctrl+C | ||
|
|
||
| Where my-release is the name of your helm release. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| dependencies: | ||
| - name: zookeeper | ||
| repository: http://storage.googleapis.com/kubernetes-charts-incubator | ||
| version: 0.1.1 | ||
| digest: sha256:8210ab022b7aaac63d45df4db7deb770bf08c5d7e24c924764648c55be5e0905 | ||
| generated: 2016-10-23T22:25:39.161124014-05:00 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| dependencies: | ||
| - name: zookeeper | ||
| version: 0.1.1 | ||
| repository: http://storage.googleapis.com/kubernetes-charts-incubator |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,129 @@ | ||
| --- | ||
| # headless service for petset DNS | ||
| apiVersion: v1 | ||
| kind: Service | ||
| metadata: | ||
| name: "{{ printf "%s-zookeeper-%s" .Release.Name .Values.zookeeper.Name | trunc 63 }}" | ||
| spec: | ||
| ports: | ||
| - port: 2181 | ||
| name: client | ||
| selector: | ||
| component: "{{.Release.Name}}-{{.Values.zookeeper.Component}}" | ||
|
|
||
| # A headless service to create DNS records | ||
| --- | ||
| apiVersion: v1 | ||
| kind: Service | ||
| metadata: | ||
| name: "{{ printf "%s-broker-%s" .Release.Name .Values.Name | trunc 63 }}" | ||
| labels: | ||
| heritage: {{.Release.Service | quote }} | ||
| release: {{.Release.Name | quote }} | ||
| chart: "{{.Chart.Name}}-{{.Chart.Version}}" | ||
| component: "{{.Release.Name}}-{{.Values.Component}}" | ||
| annotations: | ||
| "helm.sh/created": {{.Release.Time.Seconds | quote }} | ||
| service.alpha.kubernetes.io/tolerate-unready-endpoints: "true" | ||
| spec: | ||
| ports: | ||
| - port: 9092 | ||
| clusterIP: None | ||
| selector: | ||
| component: "{{.Release.Name}}-{{.Values.Component}}" | ||
| app: kafka | ||
| --- | ||
| apiVersion: v1 | ||
| kind: Service | ||
| metadata: | ||
| name: "{{ printf "%s-%s" .Release.Name .Values.Name | trunc 63 }}" | ||
| labels: | ||
| heritage: {{.Release.Service | quote }} | ||
| release: {{.Release.Name | quote }} | ||
| chart: "{{.Chart.Name}}-{{.Chart.Version}}" | ||
| component: "{{.Release.Name}}-{{.Values.Component}}" | ||
| annotations: | ||
| "helm.sh/created": {{.Release.Time.Seconds | quote }} | ||
| spec: | ||
| ports: | ||
| - port: 9092 | ||
| selector: | ||
| component: "{{.Release.Name}}-{{.Values.Component}}" | ||
| app: kafka | ||
| --- | ||
| apiVersion: apps/v1alpha1 | ||
| kind: PetSet | ||
| metadata: | ||
| name: "{{ printf "%s-%s" .Release.Name .Values.Name | trunc 63 }}" | ||
| labels: | ||
| app: kafka | ||
| heritage: {{.Release.Service | quote }} | ||
| release: {{.Release.Name | quote }} | ||
| chart: "{{.Chart.Name}}-{{.Chart.Version}}" | ||
| component: "{{.Release.Name}}-{{.Values.Component}}" | ||
| annotations: | ||
| "helm.sh/created": {{.Release.Time.Seconds | quote }} | ||
| spec: | ||
| serviceName: "{{ printf "%s-broker-%s" .Release.Name .Values.Name | trunc 63 }}" | ||
| replicas: {{ default 3 .Values.Replicas }} | ||
| template: | ||
| metadata: | ||
| labels: | ||
| app: kafka | ||
| component: "{{.Release.Name}}-{{.Values.Component}}" | ||
| annotations: | ||
| pod.alpha.kubernetes.io/initialized: "true" | ||
| pod.alpha.kubernetes.io/init-containers: '[ | ||
| ]' | ||
| spec: | ||
|
There was a problem hiding this comment. We need health check probes http://kubernetes.io/docs/user-guide/liveness/ There was a problem hiding this comment. Still needs a ready probe. We can move into incubator without. There was a problem hiding this comment. I could just add the same test for readiness as well. Not sure how valuable that would be though, in Kafka's situation. |
||
| containers: | ||
| - name: "{{ printf "%s-broker-%s" .Release.Name .Values.Name | trunc 63 }}" | ||
| image: "{{.Values.Image}}:{{.Values.ImageTag}}" | ||
| imagePullPolicy: "{{.Values.ImagePullPolicy}}" | ||
| livenessProbe: | ||
| exec: | ||
| command: | ||
| - bin/kafka-topics.sh | ||
| - --zookeeper | ||
| - "{{ printf "%s-%s" .Release.Name .Values.zookeeper.Name | trunc 63 }}:2181" | ||
| - --list | ||
| initialDelaySeconds: 30 | ||
| timeoutSeconds: 5 | ||
| name: liveness | ||
| readinessProbe: | ||
| exec: | ||
| command: | ||
| - bin/kafka-topics.sh | ||
| - --zookeeper | ||
| - "{{ printf "%s-%s" .Release.Name .Values.zookeeper.Name | trunc 63 }}:2181" | ||
| - --list | ||
| initialDelaySeconds: 30 | ||
| timeoutSeconds: 5 | ||
| name: readiness | ||
| ports: | ||
| - containerPort: 9092 | ||
| name: kafka | ||
| resources: | ||
| limits: | ||
| cpu: "{{ .Values.MaxCpu }}" | ||
| memory: "{{ .Values.MaxMemory }}" | ||
| requests: | ||
|
There was a problem hiding this comment. We need a limits section in here as well. |
||
| cpu: "{{ .Values.Cpu }}" | ||
| memory: "{{ .Values.Memory }}" | ||
| command: | ||
| - sh | ||
| - -c | ||
| - "./bin/kafka-server-start.sh config/server.properties --override zookeeper.connect={{ printf "%s-%s" .Release.Name .Values.zookeeper.Name | trunc 63 }}:2181/ --override broker.id=${HOSTNAME##*-}" | ||
| volumeMounts: | ||
| - name: datadir | ||
| mountPath: "{{ .Values.DataDirectory }}" | ||
| volumeClaimTemplates: | ||
| - metadata: | ||
| name: datadir | ||
| annotations: | ||
| volume.alpha.kubernetes.io/storage-class: anything | ||
|
There was a problem hiding this comment. Should we have this setup to do both a PVC or an empty dir? Looking for an example ... There was a problem hiding this comment. I think the empty dir can be removed. Want to leave the PVC in. I will push that out shortly. |
||
| spec: | ||
| accessModes: [ "ReadWriteOnce" ] | ||
| resources: | ||
| requests: | ||
| storage: {{.Values.Storage}} | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| Name: "kf" | ||
| Replicas: 3 | ||
| Image: "solsson/kafka" | ||
| ImageTag: "0.10.0.1" | ||
| ImagePullPolicy: "Always" | ||
| Cpu: "100m" | ||
| Memory: "512Mi" | ||
| MaxCpu: "200m" | ||
| MaxMemory: "1024Mi" | ||
| Storage: "1Gi" | ||
| DataDirectory: "/opt/kafka/data" | ||
| Component: "kafka" | ||
|
|
||
| zookeeper: | ||
| Name: "zk" | ||
| Component: "zk" |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it best practice to add proprietary file types to a global
.ignorefile? Shouldn't the user be managing these?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@kris-nova this is autogenned by helm.