Skip to content

Commit e92092b

Browse files
kasiakoziolkasiakoziol
committed
CSPL-4003 Docs update
1 parent 20d9d6c commit e92092b

File tree

1 file changed

+92
-8
lines changed

1 file changed

+92
-8
lines changed

‎docs/IndexIngestionSeparation.md‎

Lines changed: 92 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,12 @@ This separation enables:
77
- Data durability: Off‑load buffer management and retry logic to a durable message bus.
88
- Operational clarity: Separate monitoring dashboards for ingestion throughput vs indexing latency.
99

10+
# Important Note
11+
12+
> [!WARNING]
13+
> As of now, only brand new deployments are supported for Index and Ingestion Separation. No migration path is implemented, described or tested for existing deployments to move from a standard model to Index & Ingestion separation model.
14+
15+
1016
# IngestorCluster
1117

1218
IngestorCluster is introduced for high‑throughput data ingestion into a durable message bus. Its Splunk pods are configured to receive events (outputs.conf) and publish them to a message bus.
@@ -18,8 +24,8 @@ In addition to common spec inputs, the IngestorCluster resource provides the fol
1824
| Key | Type | Description |
1925
| ---------- | ------- | ------------------------------------------------- |
2026
| replicas | integer | The number of replicas (defaults to 3) |
21-
| pushBus | PushBus | Message bus configuration for publishing messages |
22-
| pipelineConfig | PipelineConfig | Configuration for pipeline |
27+
| pushBus | PushBus | Message bus configuration for publishing messages (required) |
28+
| pipelineConfig | PipelineConfig | Configuration for pipeline (required) |
2329

2430
PushBus inputs can be found in the table below. As of now, only SQS type of message bus is supported.
2531

@@ -38,9 +44,10 @@ SQS message bus inputs can be found in the table below.
3844
| largeMessageStoreEndpoint | string | AWS S3 Large Message Store endpoint (e.g. https://s3.us-west-2.amazonaws.com) |
3945
| largeMessageStorePath | string | S3 path for Large Message Store (e.g. s3://bucket-name/directory) |
4046
| deadLetterQueueName | string | Name of the SQS dead letter queue |
41-
| maxRetriesPerPart | integer | Max retries per part for retry policy max_count (The only one supported as of now) |
42-
| retryPolicy | string | Retry policy (max_retry is the only one supported as of now) |
47+
| maxRetriesPerPart | integer | Max retries per part for max_count retry policy |
48+
| retryPolicy | string | Retry policy (e.g. max_count) |
4349
| sendInterval | string | Send interval (e.g. 5s) |
50+
| encodingFormat | string | Encoding format (e.g. s2s) |
4451

4552
PipelineConfig inputs can be found in the table below.
4653

@@ -84,6 +91,7 @@ spec:
8491
maxRetriesPerPart: 4
8592
retryPolicy: max_count
8693
sendInterval: 5s
94+
encodingFormat: s2s
8795
pipelineConfig:
8896
remoteQueueRuleset: false
8997
ruleSet: true
@@ -104,8 +112,8 @@ In addition to common spec inputs, the IndexerCluster resource provides the foll
104112
| Key | Type | Description |
105113
| ---------- | ------- | ------------------------------------------------- |
106114
| replicas | integer | The number of replicas (defaults to 3) |
107-
| pullBus | PushBus | Message bus configuration for pulling messages |
108-
| pipelineConfig | PipelineConfig | Configuration for pipeline |
115+
| pullBus | PushBus | Message bus configuration for pulling messages (required) |
116+
| pipelineConfig | PipelineConfig | Configuration for pipeline (required) |
109117

110118
PullBus inputs can be found in the table below. As of now, only SQS type of message bus is supported.
111119

@@ -124,9 +132,10 @@ SQS message bus inputs can be found in the table below.
124132
| largeMessageStoreEndpoint | string | AWS S3 Large Message Store endpoint (e.g. https://s3.us-west-2.amazonaws.com) |
125133
| largeMessageStorePath | string | S3 path for Large Message Store (e.g. s3://bucket-name/directory) |
126134
| deadLetterQueueName | string | Name of SQS dead letter queue |
127-
| maxRetriesPerPart | integer | Max retries per part for retry policy max_count (The only one supported as of now) |
128-
| retryPolicy | string | Retry policy (max_retry is the only one supported as of now) |
135+
| maxRetriesPerPart | integer | Max retries per part for max_count retry policy |
136+
| retryPolicy | string | Retry policy (e.g. max_count) |
129137
| sendInterval | string | Send interval (e.g. 5s) |
138+
| encodingFormat | string | Encoding format (e.g. s2s) |
130139

131140
PipelineConfig inputs can be found in the table below.
132141

@@ -182,6 +191,7 @@ spec:
182191
maxRetriesPerPart: 4
183192
retryPolicy: max_count
184193
sendInterval: 5s
194+
encodingFormat: s2s
185195
pipelineConfig:
186196
remoteQueueRuleset: false
187197
ruleSet: true
@@ -227,6 +237,78 @@ The following additional configuration parameters may be used for all Splunk Ent
227237
| livenessInitialDelaySeconds | number | Defines initialDelaySeconds for the liveness probe |
228238
| imagePullSecrets | [ImagePullSecrets](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/) | Config to pull images from private registry (Use in conjunction with image config from [common spec](#common-spec-parameters-for-all-resources)) |
229239

240+
# Helm Charts
241+
242+
IngestorCluster can be deployed using a dedicated Helm chart. IndexerCluster helm chart has as well been enhanced to support new inputs.
243+
244+
## Example
245+
246+
Below examples describe how to define values for IngestorCluster and IndexerCluster similarly to the above yaml files specifications.
247+
248+
```
249+
ingestorCluster:
250+
enabled: true
251+
name: ingestor
252+
replicaCount: 3
253+
serviceAccount: ingestion-role-sa
254+
pipelineConfig:
255+
remoteQueueRuleset: false
256+
ruleSet: true
257+
remoteQueueTyping: false
258+
remoteQueueOutput: false
259+
typing: true
260+
indexerPipe: true
261+
pushBus:
262+
type: sqs_smartbus
263+
sqs:
264+
queueName: ing-ind-separation-q
265+
authRegion: us-west-2
266+
endpoint: https://sqs.us-west-2.amazonaws.com
267+
largeMessageStoreEndpoint: https://s3.us-west-2.amazonaws.com
268+
largeMessageStorePath: s3://ing-ind-separation/smartbus-test
269+
deadLetterQueueName: ing-ind-separation-dlq
270+
maxRetriesPerPart: 4
271+
retryPolicy: max_count
272+
sendInterval: 5s
273+
encodingFormat: s2s
274+
```
275+
276+
```
277+
clusterManager:
278+
enabled: true
279+
name: cm
280+
replicaCount: 1
281+
serviceAccount: ingestion-role-sa
282+
283+
indexerCluster:
284+
enabled: true
285+
name: indexer
286+
replicaCount: 3
287+
serviceAccount: ingestion-role-sa
288+
clusterManagerRef:
289+
name: cm
290+
pipelineConfig:
291+
remoteQueueRuleset: false
292+
ruleSet: true
293+
remoteQueueTyping: false
294+
remoteQueueOutput: false
295+
typing: true
296+
indexerPipe: true
297+
pullBus:
298+
type: sqs_smartbus
299+
sqs:
300+
queueName: ing-ind-separation-q
301+
authRegion: us-west-2
302+
endpoint: https://sqs.us-west-2.amazonaws.com
303+
largeMessageStoreEndpoint: https://s3.us-west-2.amazonaws.com
304+
largeMessageStorePath: s3://ing-ind-separation/smartbus-test
305+
deadLetterQueueName: ing-ind-separation-dlq
306+
maxRetriesPerPart: 4
307+
retryPolicy: max_count
308+
sendInterval: 5s
309+
encodingFormat: s2s
310+
```
311+
230312
# Service Account
231313

232314
To be able to configure ingestion and indexing resources correctly in a secure manner, it is required to provide these resources with the service account that is configured with a minimum set of permissions to complete required operations. With this provided, the right credentials are used by Splunk to peform its tasks.
@@ -552,6 +634,7 @@ spec:
552634
maxRetriesPerPart: 4
553635
retryPolicy: max_count
554636
sendInterval: 5s
637+
encodingFormat: s2s
555638
pipelineConfig:
556639
remoteQueueRuleset: false
557640
ruleSet: true
@@ -712,6 +795,7 @@ spec:
712795
maxRetriesPerPart: 4
713796
retryPolicy: max_count
714797
sendInterval: 5s
798+
encodingFormat: s2s
715799
pipelineConfig:
716800
remoteQueueRuleset: false
717801
ruleSet: true

0 commit comments

Comments
 (0)