Skip to content

Commit 33ef2ed

Browse files
Felix Hennigfhennig
Felix Hennig
and
fhennig
committed
Docs new landing page (#573)
# Description :computer: [preview](https://deploy-preview-374--stackable-docs.netlify.app/home/nightly/kafka/index.html) 💻 Restructured usage guide, added landing page with diagram Co-authored-by: Felix Hennig <fhennig@users.noreply.github.com>
1 parent 18f7415 commit 33ef2ed

15 files changed

+376
-376
lines changed

‎CHANGELOG.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,12 @@ All notable changes to this project will be documented in this file.
1414
- operator-rs: 0.30.1 -> 0.33.0 ([#545]).
1515
- Bumped stackable versions to "23.4.0-rc1" ([#545]).
1616
- Bumped kafka stackable versions to "23.4.0-rc2" ([#547]).
17+
- Updated landing page and restructured usage guide ([#573]).
1718

1819
[#545]: https://github.com/stackabletech/kafka-operator/pull/545
1920
[#547]: https://github.com/stackabletech/kafka-operator/pull/547
2021
[#557]: https://github.com/stackabletech/kafka-operator/pull/557
22+
[#573]: https://github.com/stackabletech/kafka-operator/pull/573
2123

2224
## [23.1.0] - 2023-01-23
2325

‎docs/modules/kafka/images/kafka_overview.drawio.svg

Lines changed: 4 additions & 0 deletions
Loading

‎docs/modules/kafka/pages/dependencies.adoc

Lines changed: 0 additions & 9 deletions
This file was deleted.

‎docs/modules/kafka/pages/getting_started/first_steps.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ include::example$getting_started/getting_started.sh[tag=install-zookeeper]
3030

3131
Create a file `kafka-znode.yaml` with the following content:
3232

33-
[source,bash]
33+
[source,yaml]
3434
----
3535
include::example$getting_started/kafka-znode.yaml[]
3636
----

‎docs/modules/kafka/pages/index.adoc

Lines changed: 41 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,50 @@
11
= Stackable Operator for Apache Kafka
2+
:description: The Stackable Operator for Apache Superset is a Kubernetes operator that can manage Apache Kafka clusters. Learn about its features, resources, dependencies and demos, and see the list of supported Kafka versions.
3+
:keywords: Stackable Operator, Apache Kafka, Kubernetes, operator, SQL, engineer, broker, big data, CRD, StatefulSet, ConfigMap, Service, Druid, ZooKeeper, NiFi, S3, demo, version
24

3-
This is an operator for Kubernetes that can manage https://kafka.apache.org/[Apache Kafka] clusters.
5+
The Stackable Operator for Apache Kafka is an operator that can deploy and manage https://kafka.apache.org/[Apache Kafka] clusters on Kubernetes.
6+
// what is Kafka?
7+
Apache Kafka is a distributed streaming platform designed to handle large volumes of data in real-time. It is commonly used for real-time data processing, data ingestion, event streaming, and messaging between applications.
8+
9+
== Getting started
10+
11+
Follow the xref:kafka:getting_started/index.adoc[] which will guide you through installing The Stackable Kafka and ZooKeeper Operators, setting up ZooKeeper and Kafka and testing your Kafka using kcat.
12+
13+
== Resources
14+
15+
The _KafkaCluster_ custom resource contains your Kafka cluster configuration. It defines a single `broker` xref:concepts:roles-and-role-groups.adoc[role].
16+
17+
image::kafka_overview.drawio.svg[A diagram depicting the Kubernetes resources created by the operator.]
18+
19+
For every xref:concepts:roles-and-role-groups.adoc#_role_groups[role group] in the `broker` role the Operator creates a StatefulSet. Multiple Services are created - one at role level, one per role group as well as one for every individual Pod - to allow access to the entire Kafka cluster, parts of it or just individual brokers.
20+
21+
For every StatefulSet (role group) a ConfigMap is deployed containing a `log4j.properties` file for xref:usage-guide/logging.adoc[logging] configuration and a `server.properties` file containing the whole Kafka configuration which is derived from the KafkaCluster resource.
22+
23+
The Operator creates a xref:concepts:service_discovery.adoc[] for the whole KafkaCluster which references the Service for the whole cluster. Other operators use this ConfigMap to connect to a Kafka cluster simply by name and it can also be used by custom third party applications to find the connection endpoint.
24+
25+
== Dependencies
26+
27+
Kafka requires xref:zookeeper:index.adoc[Apache ZooKeeper] for coordination purposes (it will not be needed in the future as it will be replaced with a https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum[built-in solution]).
28+
29+
== Connections to other products
30+
31+
Since Kafka often takes on a bridging role, many other products connect to it. In the <<demos, demos>> below you will find example data pipelines that use xref:nifi:index.adoc[Apache NiFi with the Stackable Operator] to write to Kafka and xref:nifi:index.adoc[Apache Druid with the Stackable Operator] to read from Kafka. But you can also connect using xref:spark-k8s:index.adoc[Apache Spark] or with a custom Job written in various languages.
32+
33+
== [[demos]]Demos
34+
35+
xref:stackablectl::index.adoc[] supports installing xref:stackablectl::demos/index.adoc[] with a single command. The demos are complete data piplines which showcase multiple components of the Stackable platform working together and which you can try out interactively. Both demos below inject data into Kafka using NiFi and read from the Kafka topics using Druid.
36+
37+
=== Waterlevel Demo
38+
39+
The xref:stackablectl::demos/nifi-kafka-druid-water-level-data.adoc[] demo uses data from https://www.pegelonline.wsv.de/webservice/ueberblick[PEGELONLINE] to visualize water levels in rivers and coastal regions of Germany from historic and real time data.
40+
41+
=== Earthquake Demo
42+
43+
The xref:stackablectl::demos/nifi-kafka-druid-earthquake-data.adoc[] demo ingests https://earthquake.usgs.gov/[earthquake data] into a similar pipeline as is used in the waterlevel demo.
444

5-
WARNING: This operator only works with images from the https://repo.stackable.tech/#browse/browse:docker:v2%2Fstackable%2Fkafka[Stackable] repository
645

746
== Supported Versions
847

948
The Stackable Operator for Apache Kafka currently supports the following versions of Kafka:
1049

1150
include::partial$supported-versions.adoc[]
12-
13-
== Getting the Docker image
14-
15-
[source]
16-
----
17-
docker pull docker.stackable.tech/stackable/kafka:<version>
18-
----

‎docs/modules/kafka/pages/usage-guide/configuration-environment-overrides.adoc

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
= Configuration & Environment Overrides
2+
3+
The cluster definition also supports overriding configuration properties and environment variables, either per role or per role group, where the more specific override (role group) has precedence over the less specific one (role).
4+
5+
IMPORTANT: Overriding certain properties which are set by operator (such as the ports) can interfere with the operator and can lead to problems.
6+
7+
== Configuration Properties
8+
9+
For a role or role group, at the same level of `config`, you can specify: `configOverrides` for the `server.properties`. For example, if you want to set the `auto.create.topics.enable` to disable automatic topic creation, it can be configured in the `KafkaCluster` resource like so:
10+
11+
[source,yaml]
12+
----
13+
brokers:
14+
roleGroups:
15+
default:
16+
configOverrides:
17+
server.properties:
18+
auto.create.topics.enable: "false"
19+
replicas: 1
20+
----
21+
22+
Just as for the `config`, it is possible to specify this at role level as well:
23+
24+
[source,yaml]
25+
----
26+
brokers:
27+
configOverrides:
28+
server.properties:
29+
auto.create.topics.enable: "false"
30+
roleGroups:
31+
default:
32+
replicas: 1
33+
----
34+
35+
All override property values must be strings.
36+
37+
For a full list of configuration options we refer to the Apache Kafka https://kafka.apache.org/documentation/#configuration[Configuration Reference].
38+
39+
== Environment Variables
40+
41+
In a similar fashion, environment variables can be (over)written. For example per role group:
42+
43+
[source,yaml]
44+
----
45+
servers:
46+
roleGroups:
47+
default:
48+
envOverrides:
49+
MY_ENV_VAR: "MY_VALUE"
50+
replicas: 1
51+
----
52+
53+
or per role:
54+
55+
[source,yaml]
56+
----
57+
servers:
58+
envOverrides:
59+
MY_ENV_VAR: "MY_VALUE"
60+
roleGroups:
61+
default:
62+
replicas: 1
63+
----

‎docs/modules/kafka/pages/discovery.adoc renamed to ‎docs/modules/kafka/pages/usage-guide/discovery.adoc

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,9 @@
33
:brokerPort: 9092
44

55
= Discovery
6+
:page-aliases: discovery.adoc
67

7-
The Stackable Operator for Apache Kafka publishes a discovery https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#configmap-v1-core[`ConfigMap`], which exposes a client configuration bundle that allows access to the Apache Kafka cluster.
8+
The Stackable Operator for Apache Kafka publishes a xref:concepts:service_discovery.adoc[service discovery ConfigMap] which exposes a client configuration bundle that allows access to the Apache Kafka cluster.
89

910
The bundle includes a thrift connection string to access the Kafka broker service. This string may be used by other operators or tools to configure their products with access to Kafka. This is limited to internal cluster access.
1011

@@ -22,14 +23,14 @@ metadata:
2223
spec:
2324
[...]
2425
----
25-
<1> The name of the Kafka cluster, which is also the name of the created discovery `ConfigMap`.
26-
<2> The namespace of the discovery `ConfigMap`.
26+
<1> The name of the Kafka cluster, which is also the name of the created discovery ConfigMap.
27+
<2> The namespace of the discovery ConfigMap.
2728

28-
The resulting discovery `ConfigMap` is `{namespace}/{clusterName}`.
29+
The resulting discovery ConfigMap is `{namespace}/{clusterName}`.
2930

3031
== Contents
3132

32-
The `{namespace}/{clusterName}` discovery `ConfigMap` contains the following fields where `{clusterName}` represents the name and `{namespace}` the namespace of the cluster:
33+
The `{namespace}/{clusterName}` discovery ConfigMap contains the following fields where `{clusterName}` represents the name and `{namespace}` the namespace of the cluster:
3334

3435
`KAFKA`::
3536
====

‎docs/modules/kafka/pages/usage-guide/index.adoc

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
= Usage guide
2+
:page-aliases: usage.adoc

‎docs/modules/kafka/pages/usage-guide/logging.adoc

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
= Log aggregation
2+
3+
The logs can be forwarded to a Vector log aggregator by providing a discovery
4+
ConfigMap for the aggregator and by enabling the log agent:
5+
6+
[source,yaml]
7+
----
8+
spec:
9+
clusterConfig:
10+
vectorAggregatorConfigMapName: vector-aggregator-discovery
11+
brokers:
12+
config:
13+
logging:
14+
enableVectorAgent: true
15+
----
16+
17+
Further information on how to configure logging, can be found in
18+
xref:home:concepts:logging.adoc[].

‎docs/modules/kafka/pages/usage-guide/monitoring.adoc

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
= Monitoring
2+
3+
The managed Kafka instances are automatically configured to export Prometheus metrics. See
4+
xref:home:operators:monitoring.adoc[] for more details.

‎docs/modules/kafka/pages/usage-guide/pod-placement.adoc

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
= Pod Placement
2+
3+
You can configure Pod placement for Kafka brokers as described in xref:concepts:pod_placement.adoc[].
4+
5+
By default, the operator configures the following Pod placement constraints:
6+
7+
[source,yaml]
8+
----
9+
affinity:
10+
podAntiAffinity:
11+
preferredDuringSchedulingIgnoredDuringExecution:
12+
- podAffinityTerm:
13+
labelSelector:
14+
matchLabels:
15+
app.kubernetes.io/component: broker
16+
app.kubernetes.io/instance: cluster-name
17+
app.kubernetes.io/name: kafka
18+
topologyKey: kubernetes.io/hostname
19+
weight: 70
20+
----
21+
22+
In the example above `cluster-name` is the name of the Kafka custom resource that owns this Pod.

‎docs/modules/kafka/pages/usage-guide/security.adoc

Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
= Security
2+
3+
== Encryption
4+
5+
The internal and client communication can be encrypted TLS. This requires the xref:secret-operator::index.adoc[Secret Operator] to be present in order to provide certificates. The utilized certificates can be changed in a top-level config.
6+
7+
[source,yaml]
8+
----
9+
---
10+
apiVersion: kafka.stackable.tech/v1alpha1
11+
kind: KafkaCluster
12+
metadata:
13+
name: simple-kafka
14+
spec:
15+
image:
16+
productVersion: 3.3.1
17+
stackableVersion: "23.4.0-rc2"
18+
clusterConfig:
19+
zookeeperConfigMapName: simple-kafka-znode
20+
tls:
21+
serverSecretClass: tls # <1>
22+
internalSecretClass: kafka-internal-tls # <2>
23+
brokers:
24+
roleGroups:
25+
default:
26+
replicas: 3
27+
----
28+
<1> The `spec.clusterConfig.tls.serverSecretClass` refers to the client-to-server encryption. Defaults to the `tls` secret. Can be deactivated by setting `serverSecretClass` to `null`.
29+
<2> The `spec.clusterConfig.tls.internalSecretClass` refers to the broker-to-broker internal encryption. This must be explicitly set or defaults to `tls`. May be disabled by setting `internalSecretClass` to `null`.
30+
31+
The `tls` secret is deployed from the xref:secret-operator::index.adoc[Secret Operator] and looks like this:
32+
33+
[source,yaml]
34+
----
35+
---
36+
apiVersion: secrets.stackable.tech/v1alpha1
37+
kind: SecretClass
38+
metadata:
39+
name: tls
40+
spec:
41+
backend:
42+
autoTls:
43+
ca:
44+
secret:
45+
name: secret-provisioner-tls-ca
46+
namespace: default
47+
autoGenerate: true
48+
----
49+
50+
You can create your own secrets and reference them e.g. in the `spec.clusterConfig.tls.serverSecretClass` or `spec.clusterConfig.tls.internalSecretClass` to use different certificates.
51+
52+
== Authentication
53+
54+
The internal or broker-to-broker communication is authenticated via TLS. In order to enforce TLS authentication for client-to-server communication, you can set an `AuthenticationClass` reference in the custom resource provided by the xref:commons-operator::index.adoc[Commons Operator].
55+
56+
[source,yaml]
57+
----
58+
---
59+
apiVersion: authentication.stackable.tech/v1alpha1
60+
kind: AuthenticationClass
61+
metadata:
62+
name: kafka-client-tls # <2>
63+
spec:
64+
provider:
65+
tls:
66+
clientCertSecretClass: kafka-client-auth-secret # <3>
67+
---
68+
apiVersion: secrets.stackable.tech/v1alpha1
69+
kind: SecretClass
70+
metadata:
71+
name: kafka-client-auth-secret # <4>
72+
spec:
73+
backend:
74+
autoTls:
75+
ca:
76+
secret:
77+
name: secret-provisioner-tls-kafka-client-ca
78+
namespace: default
79+
autoGenerate: true
80+
---
81+
apiVersion: kafka.stackable.tech/v1alpha1
82+
kind: KafkaCluster
83+
metadata:
84+
name: simple-kafka
85+
spec:
86+
image:
87+
productVersion: 3.3.1
88+
stackableVersion: "23.4.0-rc2"
89+
clusterConfig:
90+
authentication:
91+
- authenticationClass: kafka-client-tls # <1>
92+
zookeeperConfigMapName: simple-kafka-znode
93+
brokers:
94+
roleGroups:
95+
default:
96+
replicas: 3
97+
----
98+
<1> The `clusterConfig.authentication.authenticationClass` can be set to use TLS for authentication. This is optional.
99+
<2> The referenced `AuthenticationClass` that references a `SecretClass` to provide certificates.
100+
<3> The reference to a `SecretClass`.
101+
<4> The `SecretClass` that is referenced by the `AuthenticationClass` in order to provide certificates.
102+
103+
104+
== [[authorization]]Authorization
105+
106+
If you wish to include integration with xref:opa::index.adoc[Open Policy Agent] and already have an OPA cluster, then you can include an `opa` field pointing to the OPA cluster discovery `ConfigMap` and the required package. The package is optional and will default to the `metadata.name` field:
107+
108+
[source,yaml]
109+
----
110+
---
111+
apiVersion: kafka.stackable.tech/v1alpha1
112+
kind: KafkaCluster
113+
metadata:
114+
name: simple-kafka
115+
spec:
116+
image:
117+
productVersion: 3.3.1
118+
stackableVersion: "23.4.0-rc2"
119+
clusterConfig:
120+
authorization:
121+
opa:
122+
configMapName: simple-opa
123+
package: kafka
124+
zookeeperConfigMapName: simple-kafka-znode
125+
brokers:
126+
roleGroups:
127+
default:
128+
replicas: 1
129+
----
130+
131+
You can change some opa cache properties by overriding:
132+
133+
[source,yaml]
134+
----
135+
---
136+
apiVersion: kafka.stackable.tech/v1alpha1
137+
kind: KafkaCluster
138+
metadata:
139+
name: simple-kafka
140+
spec:
141+
image:
142+
productVersion: 3.3.1
143+
stackableVersion: "23.4.0-rc2"
144+
clusterConfig:
145+
authorization:
146+
opa:
147+
configMapName: simple-opa
148+
package: kafka
149+
zookeeperConfigMapName: simple-kafka-znode
150+
brokers:
151+
configOverrides:
152+
server.properties:
153+
opa.authorizer.cache.initial.capacity: "100"
154+
opa.authorizer.cache.maximum.size: "100"
155+
opa.authorizer.cache.expire.after.seconds: "10"
156+
roleGroups:
157+
default:
158+
replicas: 1
159+
----
160+
161+
A full list of settings and their respective defaults can be found https://github.com/anderseknert/opa-kafka-plugin[here].

0 commit comments

Comments
 (0)