Skip to content

CSPL-3972 Addressing TODOs #1576

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 10 commits into
base: CSPL-3551-ingestion-cr
Choose a base branch
from
Open

CSPL-3972 Addressing TODOs #1576

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
2af9061
CSPL-3972 Addressing TODOs
kasiakoziol Sep 11, 2025
053352d
CSPL-3551 Moving default from types to controller code
kasiakoziol Sep 11, 2025
e844286
CSPL-3551 Applying previous changes to IndexerCluster
kasiakoziol Sep 18, 2025
500cdfa
CSPL-3551 Moving validations to separate function and adding validati…
kasiakoziol Sep 19, 2025
4b064a6
CSPL-3551 Making sure all inputs are put into status
kasiakoziol Sep 19, 2025
9c6cd69
CSPL-3551 Update of conf files when queue name or type change
kasiakoziol Sep 22, 2025
759b09b
CSPL-3551 Unit tests
kasiakoziol Sep 23, 2025
20d9d6c
CSPL-4003 Helm charts support for bus and pipeline configs
kasiakoziol Sep 29, 2025
f9fb45a
CSPL-4003 Docs update
kasiakoziol Oct 1, 2025
c5e4c2b
CSPL-4003 Addressing comments
kasiakoziol Oct 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions api/v4/indexercluster_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,12 @@ type IndexerClusterStatus struct {
// status of each indexer cluster peer
Peers []IndexerClusterMemberStatus `json:"peers"`

// Pipeline configuration status
PipelineConfig PipelineConfigSpec `json:"pipelineConfig,omitempty"`

// Pull Bus status
PullBus PushBusSpec `json:"pullBus,omitempty"`

// Auxillary message describing CR status
Message string `json:"message"`
}
Expand Down
10 changes: 9 additions & 1 deletion api/v4/ingestorcluster_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,8 @@ type SQSSpec struct {
RetryPolicy string `json:"retryPolicy"`

SendInterval string `json:"sendInterval"`

EncodingFormat string `json:"encodingFormat"`
}

type PipelineConfigSpec struct {
Expand All @@ -88,7 +90,7 @@ type PipelineConfigSpec struct {

Typing bool `json:"typing"`

IndexerPipe bool `json:"indexerPipe,omitempty"`
IndexerPipe bool `json:"indexerPipe"`
}

// IngestorClusterStatus defines the observed state of Ingestor Cluster
Expand Down Expand Up @@ -116,6 +118,12 @@ type IngestorClusterStatus struct {

// Auxillary message describing CR status
Message string `json:"message"`

// Pipeline configuration status
PipelineConfig PipelineConfigSpec `json:"pipelineConfig"`

// Push Bus status
PushBus PushBusSpec `json:"pushBus"`
}

// +kubebuilder:object:root=true
Expand Down
4 changes: 4 additions & 0 deletions api/v4/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

47 changes: 47 additions & 0 deletions config/crd/bases/enterprise.splunk.com_indexerclusters.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5630,6 +5630,8 @@ spec:
type: string
deadLetterQueueName:
type: string
encodingFormat:
type: string
endpoint:
type: string
largeMessageStoreEndpoint:
Expand Down Expand Up @@ -8379,6 +8381,51 @@ spec:
- Terminating
- Error
type: string
pipelineConfig:
description: Pipeline configuration status
properties:
indexerPipe:
type: boolean
remoteQueueOutput:
type: boolean
remoteQueueRuleset:
type: boolean
remoteQueueTyping:
type: boolean
ruleSet:
type: boolean
typing:
type: boolean
type: object
pullBus:
description: Pull Bus status
properties:
sqs:
properties:
authRegion:
type: string
deadLetterQueueName:
type: string
encodingFormat:
type: string
endpoint:
type: string
largeMessageStoreEndpoint:
type: string
largeMessageStorePath:
type: string
maxRetriesPerPart:
type: integer
queueName:
type: string
retryPolicy:
type: string
sendInterval:
type: string
type: object
type:
type: string
type: object
readyReplicas:
description: current number of ready indexer peers
format: int32
Expand Down
47 changes: 47 additions & 0 deletions config/crd/bases/enterprise.splunk.com_ingestorclusters.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1605,6 +1605,8 @@ spec:
type: string
deadLetterQueueName:
type: string
encodingFormat:
type: string
endpoint:
type: string
largeMessageStoreEndpoint:
Expand Down Expand Up @@ -4559,6 +4561,51 @@ spec:
- Terminating
- Error
type: string
pipelineConfig:
description: Pipeline configuration status
properties:
indexerPipe:
type: boolean
remoteQueueOutput:
type: boolean
remoteQueueRuleset:
type: boolean
remoteQueueTyping:
type: boolean
ruleSet:
type: boolean
typing:
type: boolean
type: object
pushBus:
description: Push Bus status
properties:
sqs:
properties:
authRegion:
type: string
deadLetterQueueName:
type: string
encodingFormat:
type: string
endpoint:
type: string
largeMessageStoreEndpoint:
type: string
largeMessageStorePath:
type: string
maxRetriesPerPart:
type: integer
queueName:
type: string
retryPolicy:
type: string
sendInterval:
type: string
type: object
type:
type: string
type: object
readyReplicas:
description: Number of ready ingestor pods
format: int32
Expand Down
6 changes: 4 additions & 2 deletions config/samples/enterprise_v4_ingestorcluster.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,7 @@ apiVersion: enterprise.splunk.com/v4
kind: IngestorCluster
metadata:
name: ingestorcluster-sample
spec:
# TODO(user): Add fields here
finalizers:
- "enterprise.splunk.com/delete-pvc"
spec: {}
# TODO(user): Add fields here
100 changes: 92 additions & 8 deletions docs/IndexIngestionSeparation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,12 @@ This separation enables:
- Data durability: Off‑load buffer management and retry logic to a durable message bus.
- Operational clarity: Separate monitoring dashboards for ingestion throughput vs indexing latency.

# Important Note

> [!WARNING]
> **As of now, only brand new deployments are supported for Index and Ingestion Separation. No migration path is implemented, described or tested for existing deployments to move from a standard model to Index & Ingestion separation model.**


# IngestorCluster

IngestorCluster is introduced for high‑throughput data ingestion into a durable message bus. Its Splunk pods are configured to receive events (outputs.conf) and publish them to a message bus.
Expand All @@ -18,8 +24,8 @@ In addition to common spec inputs, the IngestorCluster resource provides the fol
| Key | Type | Description |
| ---------- | ------- | ------------------------------------------------- |
| replicas | integer | The number of replicas (defaults to 3) |
| pushBus | PushBus | Message bus configuration for publishing messages |
| pipelineConfig | PipelineConfig | Configuration for pipeline |
| pushBus | PushBus | Message bus configuration for publishing messages (required) |
| pipelineConfig | PipelineConfig | Configuration for pipeline (required) |

PushBus inputs can be found in the table below. As of now, only SQS type of message bus is supported.

Expand All @@ -38,9 +44,10 @@ SQS message bus inputs can be found in the table below.
| largeMessageStoreEndpoint | string | AWS S3 Large Message Store endpoint (e.g. https://s3.us-west-2.amazonaws.com) |
| largeMessageStorePath | string | S3 path for Large Message Store (e.g. s3://bucket-name/directory) |
| deadLetterQueueName | string | Name of the SQS dead letter queue |
| maxRetriesPerPart | integer | Max retries per part for retry policy max_count (The only one supported as of now) |
| retryPolicy | string | Retry policy (max_retry is the only one supported as of now) |
| maxRetriesPerPart | integer | Max retries per part for max_count retry policy |
| retryPolicy | string | Retry policy (e.g. max_count) |
| sendInterval | string | Send interval (e.g. 5s) |
| encodingFormat | string | Encoding format (e.g. s2s) |

PipelineConfig inputs can be found in the table below.

Expand Down Expand Up @@ -84,6 +91,7 @@ spec:
maxRetriesPerPart: 4
retryPolicy: max_count
sendInterval: 5s
encodingFormat: s2s
pipelineConfig:
remoteQueueRuleset: false
ruleSet: true
Expand All @@ -104,8 +112,8 @@ In addition to common spec inputs, the IndexerCluster resource provides the foll
| Key | Type | Description |
| ---------- | ------- | ------------------------------------------------- |
| replicas | integer | The number of replicas (defaults to 3) |
| pullBus | PushBus | Message bus configuration for pulling messages |
| pipelineConfig | PipelineConfig | Configuration for pipeline |
| pullBus | PushBus | Message bus configuration for pulling messages (required) |
| pipelineConfig | PipelineConfig | Configuration for pipeline (required) |

PullBus inputs can be found in the table below. As of now, only SQS type of message bus is supported.

Expand All @@ -124,9 +132,10 @@ SQS message bus inputs can be found in the table below.
| largeMessageStoreEndpoint | string | AWS S3 Large Message Store endpoint (e.g. https://s3.us-west-2.amazonaws.com) |
| largeMessageStorePath | string | S3 path for Large Message Store (e.g. s3://bucket-name/directory) |
| deadLetterQueueName | string | Name of SQS dead letter queue |
| maxRetriesPerPart | integer | Max retries per part for retry policy max_count (The only one supported as of now) |
| retryPolicy | string | Retry policy (max_retry is the only one supported as of now) |
| maxRetriesPerPart | integer | Max retries per part for max_count retry policy |
| retryPolicy | string | Retry policy (e.g. max_count) |
| sendInterval | string | Send interval (e.g. 5s) |
| encodingFormat | string | Encoding format (e.g. s2s) |

PipelineConfig inputs can be found in the table below.

Expand Down Expand Up @@ -182,6 +191,7 @@ spec:
maxRetriesPerPart: 4
retryPolicy: max_count
sendInterval: 5s
encodingFormat: s2s
pipelineConfig:
remoteQueueRuleset: false
ruleSet: true
Expand Down Expand Up @@ -227,6 +237,78 @@ The following additional configuration parameters may be used for all Splunk Ent
| livenessInitialDelaySeconds | number | Defines initialDelaySeconds for the liveness probe |
| imagePullSecrets | [ImagePullSecrets](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/) | Config to pull images from private registry (Use in conjunction with image config from [common spec](#common-spec-parameters-for-all-resources)) |

# Helm Charts

An IngestorCluster template has been added to the splunk/splunk-enterprise Helm chart. The IndexerCluster template has also been enhanced to support new inputs.

## Example

Below examples describe how to define values for IngestorCluster and IndexerCluster similarly to the above yaml files specifications.

```
ingestorCluster:
enabled: true
name: ingestor
replicaCount: 3
serviceAccount: ingestion-role-sa
pipelineConfig:
remoteQueueRuleset: false
ruleSet: true
remoteQueueTyping: false
remoteQueueOutput: false
typing: true
indexerPipe: true
pushBus:
type: sqs_smartbus
sqs:
queueName: ing-ind-separation-q
authRegion: us-west-2
endpoint: https://sqs.us-west-2.amazonaws.com
largeMessageStoreEndpoint: https://s3.us-west-2.amazonaws.com
largeMessageStorePath: s3://ing-ind-separation/smartbus-test
deadLetterQueueName: ing-ind-separation-dlq
maxRetriesPerPart: 4
retryPolicy: max_count
sendInterval: 5s
encodingFormat: s2s
```

```
clusterManager:
enabled: true
name: cm
replicaCount: 1
serviceAccount: ingestion-role-sa

indexerCluster:
enabled: true
name: indexer
replicaCount: 3
serviceAccount: ingestion-role-sa
clusterManagerRef:
name: cm
pipelineConfig:
remoteQueueRuleset: false
ruleSet: true
remoteQueueTyping: false
remoteQueueOutput: false
typing: true
indexerPipe: true
pullBus:
type: sqs_smartbus
sqs:
queueName: ing-ind-separation-q
authRegion: us-west-2
endpoint: https://sqs.us-west-2.amazonaws.com
largeMessageStoreEndpoint: https://s3.us-west-2.amazonaws.com
largeMessageStorePath: s3://ing-ind-separation/smartbus-test
deadLetterQueueName: ing-ind-separation-dlq
maxRetriesPerPart: 4
retryPolicy: max_count
sendInterval: 5s
encodingFormat: s2s
```

# Service Account

To be able to configure ingestion and indexing resources correctly in a secure manner, it is required to provide these resources with the service account that is configured with a minimum set of permissions to complete required operations. With this provided, the right credentials are used by Splunk to peform its tasks.
Expand Down Expand Up @@ -552,6 +634,7 @@ spec:
maxRetriesPerPart: 4
retryPolicy: max_count
sendInterval: 5s
encodingFormat: s2s
pipelineConfig:
remoteQueueRuleset: false
ruleSet: true
Expand Down Expand Up @@ -712,6 +795,7 @@ spec:
maxRetriesPerPart: 4
retryPolicy: max_count
sendInterval: 5s
encodingFormat: s2s
pipelineConfig:
remoteQueueRuleset: false
ruleSet: true
Expand Down
Loading
Loading