Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[receiver/kafka]: support receiving from multiple topics #33204

Closed
yilmazo opened this issue May 23, 2024 · 4 comments
Closed

[receiver/kafka]: support receiving from multiple topics #33204

yilmazo opened this issue May 23, 2024 · 4 comments
Labels

Comments

@yilmazo
Copy link

yilmazo commented May 23, 2024

Component(s)

receiver/kafka

Is your feature request related to a problem? Please describe.

I want to collect logs from multiple Kubernetes clusters using opentelemetry collector, and send them to elasticsearch.
My setup briefly looks like this:
collectors (in k8s clusters) --> kafka --> collector (centralized) --> elasticsearch

I want to use multiple topics on kafka in order to mitigate noisy neighbor situations.
topic_from_attribute configuration on kafka exporter helps me to send logs to separate topics easily, but there is no way to provide a topic pattern or regex for consuming from multiple topics. As a result, I need to automate the process of adding/removing kafka receivers on my centralized collector's config and I'm not sure that it will be a problem to add ~500 kafka receivers on collector config.

Describe the solution you'd like

Instead of this:

receivers:
  kafka/1:
    topic: logs_1
  kafka/2:
    topic: logs_2
  kafka/3:
    topic: logs_3

I want to be able to provide a pattern or regexp.

receivers:
  kafka:
    topicPattern: logs_*

Note: Collector should also periodically check all topics and update the listened topic list based on the topicPattern.

Describe alternatives you've considered

I'm open to any other solutions or recommendations.

Additional context

No response

@yilmazo yilmazo added enhancement New feature or request needs triage New item requiring triage labels May 23, 2024
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@ChrsMark
Copy link
Member

Could be related to #32735.

Copy link
Contributor

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

Copy link
Contributor

This issue has been closed as inactive because it has been stale for 120 days with no activity.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Sep 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants