FEATURE: Kafka split events into different topics based on event.module

[reyesj2]

This functionality existed and was later removed. With upgrade to Elasticsearch 8.17.x this capability was added back.

Splitting topics by event.module would allow external consumers to ingest only the events relevant to them.

[reyesj2]

While in 8.17.2 the setting for splitting event data among more than 1 default topic was added back. Certain functionality was left out.

You can split data by a single field eg. data_stream.dataset. However, that means every different integration /event type creates a new topic.

Example of using data_stream.dataset

[root@jr-dev-kafka-man onion]# so-kafka-cli kafka-topics.sh --bootstrap-server 10.250.0.238:9092 --command-config /opt/kafka/config/kraft/client.properties --list
default-securityonion
elastic_agent
elastic_agent.elastic_agent
elastic_agent.endpoint_security
elastic_agent.filebeat
elastic_agent.filebeat_input
elastic_agent.fleet_server
elastic_agent.metricbeat
elastic_agent.osquerybeat
elasticsearch.server
endpoint.events.api
endpoint.events.file
endpoint.events.library
endpoint.events.network
endpoint.events.process
endpoint.events.registry
endpoint.events.security
endpoint.metadata
endpoint.metrics
endpoint.policy
kratos
soc
strelka
suricata
syslog
system.application
system.auth
system.security
system.syslog
system.system
winlog.winlog
zeek