Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[reciever/kafkametrics] Add support for topic configurations and assigning cluster alias #34148

Closed
RutvikS-crest opened this issue Jul 18, 2024 · 3 comments
Labels
enhancement New feature or request needs triage New item requiring triage receiver/kafkametrics

Comments

@RutvikS-crest
Copy link
Contributor

RutvikS-crest commented Jul 18, 2024

Component(s)

receiver/kafkametrics

Is your feature request related to a problem? Please describe.

kafkametrics reciever only supports one topic metric.

  • kafka.topic.partitions

We want to monitor multiple kafka clusters, It will be easier if there is an additional metric dimension for each cluster.

We want to monitor kafka's topic configurations like replication-factor, min-insync-relicas, log-retention settings etc. These metrics will help us monitor and standardise topic configurations across all of our clusters.

Describe the solution you'd like

I propose to support following metrics that can easily be fetched using sarama client.

  • kafka.topic.replication_factor
  • kafka.topic.log_retention_ms
  • kafka.topic.log_retention_bytes
  • kafka.topic.min_insync_replicas

Add an optional configuration field to assign cluster alias name.

receivers:
  kafkametrics:
    protocol_version: 2.0.0
    cluster_alias: kafka-ecom
    scrapers:
      - brokers
      - topics
      - consumers

Describe alternatives you've considered

No response

Additional context

No response

@RutvikS-crest RutvikS-crest added enhancement New feature or request needs triage New item requiring triage labels Jul 18, 2024
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@RutvikS-crest RutvikS-crest changed the title kafkametrics reciever | Add support for topic configurations [reciever/kafkametrics] Add support for topic configurations Jul 18, 2024
@RutvikS-crest RutvikS-crest changed the title [reciever/kafkametrics] Add support for topic configurations [reciever/kafkametrics] Add support for topic configurations and assigning cluster alias Jul 18, 2024
dmitryax added a commit that referenced this issue Aug 9, 2024
…ter_alias field (#34149)

**Description:**
1. Adding support for kafka topic and broker configurations. receiver
will expose following metrics.
- `kafka.broker.log_retention_period`
- `kafka.topic.replication_factor`
- `kafka.topic.log_retention_period`
- `kafka.topic.log_retention_size`
- `kafka.topic.min_insync_replicas`

2. Add an optional configuration `cluster_alias` field. 
3. assign a new metric attribute `cluster_alias` to all metrics.

**Link to tracking Issue:** #34148 

---------

Co-authored-by: Curtis Robert <crobert@splunk.com>
Co-authored-by: Dmitrii Anoshin <anoshindx@gmail.com>
@crobert-1
Copy link
Member

Hello @RutvikS-crest, was this completed in #34149, or is there more work to do here? If it's complete, we can close the issue.

@RutvikS-crest
Copy link
Contributor Author

Hi @crobert-1, Let's close this issue, It's completed.

f7o pushed a commit to f7o/opentelemetry-collector-contrib that referenced this issue Sep 12, 2024
…ter_alias field (open-telemetry#34149)

**Description:**
1. Adding support for kafka topic and broker configurations. receiver
will expose following metrics.
- `kafka.broker.log_retention_period`
- `kafka.topic.replication_factor`
- `kafka.topic.log_retention_period`
- `kafka.topic.log_retention_size`
- `kafka.topic.min_insync_replicas`

2. Add an optional configuration `cluster_alias` field. 
3. assign a new metric attribute `cluster_alias` to all metrics.

**Link to tracking Issue:** open-telemetry#34148 

---------

Co-authored-by: Curtis Robert <crobert@splunk.com>
Co-authored-by: Dmitrii Anoshin <anoshindx@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request needs triage New item requiring triage receiver/kafkametrics
Projects
None yet
Development

No branches or pull requests

2 participants