Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[receiver/statsd] Add Option to Aggregate on IP/Host #34851

Merged
merged 14 commits into from
Sep 25, 2024

Conversation

daidokoro
Copy link
Contributor

Description:

The statsdreceiver only aggregates metrics on protocol+host+ip, this leads to issues or inconsistencies when dealing with clients that constantly switch tcp/udp ports. To address the issue, this PR adds a configuration option enableIPOnlyAggregation that allows the use to specify if they want to aggregate on the IP instead of IP+Port.

For example:

otel_config.yaml:

receivers:
  statsd:
    endpoint: "0.0.0.0:8125"
    enable_metric_type: true
    is_monotonic_counter: false
    aggregation_interval: 10s
    enable_ip_only_aggregation: true # <-- enable ip only aggregation
    timer_histogram_mapping:
      - statsd_type: "timing"
        observer_type: "histogram"
        histogram: 
          max_size: 50  

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    metrics:
      receivers:
        - statsd
      exporters:
        - debug

run:

STATSD_HOST="localhost"
STATSD_PORT=8125

for port in {10000..10010}; do
  echo -n "my.metric:1|c" | nc -w 1 -u -p $port ${STATSD_HOST} ${STATSD_PORT}
  echo "Sent from port $port"
done

result:

2024-08-26T23:36:00.224+0200    info    ResourceMetrics #0
Resource SchemaURL: 
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope otelcol/statsdreceiver 0.103.0-dev
Metric #0
Descriptor:
     -> Name: -n my.metric
     -> Description: 
     -> Unit: 
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Delta
NumberDataPoints #0
Data point attributes:
     -> metric_type: Str(counter)
StartTimestamp: 2024-08-26 21:35:50.223101 +0000 UTC
Timestamp: 2024-08-26 21:36:00.224252 +0000 UTC
Value: 7
        {"kind": "exporter", "data_type": "metrics", "name": "debug"}
2024-08-26T23:36:10.224+0200    info    MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 1, "data points": 1}
2024-08-26T23:36:10.224+0200    info    ResourceMetrics #0
Resource SchemaURL: 
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope otelcol/statsdreceiver 0.103.0-dev
Metric #0
Descriptor:
     -> Name: -n my.metric
     -> Description: 
     -> Unit: 
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Delta
NumberDataPoints #0
Data point attributes:
     -> metric_type: Str(counter)
StartTimestamp: 2024-08-26 21:36:00.224252 +0000 UTC
Timestamp: 2024-08-26 21:36:10.224607 +0000 UTC
Value: 4
        {"kind": "exporter", "data_type": "metrics", "name": "debug"}

Instead of generating 11 metrics for each port that was used to send, only 2 metrics are blocks are returned, who's values total 11.

2024-08-26 23 44 15

Link to tracking Issue: #23809

Testing:

  • Added unit tests

Documentation:

  • Added information to the statsdreceiver README.md describing the option.

@daidokoro daidokoro requested a review from a team August 26, 2024 21:50
@github-actions github-actions bot added the receiver/statsd statsd related issues label Aug 26, 2024
Copy link
Contributor

@atoulme atoulme left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, needs codeowner review.

@dmitryax dmitryax added the ready to merge Code review completed; ready to merge by maintainers label Sep 11, 2024
@daidokoro daidokoro requested a review from a team as a code owner September 22, 2024 21:07
@dmitryax dmitryax merged commit 5ad5841 into open-telemetry:main Sep 25, 2024
162 checks passed
@github-actions github-actions bot added this to the next release milestone Sep 25, 2024
@povilasv povilasv deleted the es-337 branch September 25, 2024 07:55
jriguera pushed a commit to springernature/opentelemetry-collector-contrib that referenced this pull request Oct 4, 2024
…34851)

**Description:** <Describe what has changed.>

The `statsdreceiver` only aggregates metrics on `protocol+host+ip`, this
leads to issues or inconsistencies when dealing with clients that
constantly switch tcp/udp ports. To address the issue, this PR adds a
configuration option `enableIPOnlyAggregation` that allows the use to
specify if they want to aggregate on the `IP` instead of `IP+Port`.

For example:

_otel_config.yaml:_

```yaml
receivers:
  statsd:
    endpoint: "0.0.0.0:8125"
    enable_metric_type: true
    is_monotonic_counter: false
    aggregation_interval: 10s
    enable_ip_only_aggregation: true # <-- enable ip only aggregation
    timer_histogram_mapping:
      - statsd_type: "timing"
        observer_type: "histogram"
        histogram: 
          max_size: 50  

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    metrics:
      receivers:
        - statsd
      exporters:
        - debug
```

_run:_
```sh
STATSD_HOST="localhost"
STATSD_PORT=8125

for port in {10000..10010}; do
  echo -n "my.metric:1|c" | nc -w 1 -u -p $port ${STATSD_HOST} ${STATSD_PORT}
  echo "Sent from port $port"
done
```

_result:_
```
2024-08-26T23:36:00.224+0200    info    ResourceMetrics #0
Resource SchemaURL: 
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope otelcol/statsdreceiver 0.103.0-dev
Metric #0
Descriptor:
     -> Name: -n my.metric
     -> Description: 
     -> Unit: 
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Delta
NumberDataPoints #0
Data point attributes:
     -> metric_type: Str(counter)
StartTimestamp: 2024-08-26 21:35:50.223101 +0000 UTC
Timestamp: 2024-08-26 21:36:00.224252 +0000 UTC
Value: 7
        {"kind": "exporter", "data_type": "metrics", "name": "debug"}
2024-08-26T23:36:10.224+0200    info    MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 1, "data points": 1}
2024-08-26T23:36:10.224+0200    info    ResourceMetrics #0
Resource SchemaURL: 
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope otelcol/statsdreceiver 0.103.0-dev
Metric #0
Descriptor:
     -> Name: -n my.metric
     -> Description: 
     -> Unit: 
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Delta
NumberDataPoints #0
Data point attributes:
     -> metric_type: Str(counter)
StartTimestamp: 2024-08-26 21:36:00.224252 +0000 UTC
Timestamp: 2024-08-26 21:36:10.224607 +0000 UTC
Value: 4
        {"kind": "exporter", "data_type": "metrics", "name": "debug"}
```

Instead of generating 11 metrics for each port that was used to send,
only 2 metrics are blocks are returned, who's values total 11.

![2024-08-26 23 44
15](https://github.com/user-attachments/assets/6b8a89d1-186e-4257-9c82-90c5f9d14f98)


**Link to tracking Issue:** open-telemetry#23809

**Testing:** 

- [x] Added unit tests

**Documentation:** <Describe the documentation added.>

- [x] Added information to the statsdreceiver `README.md` describing the
option.

---------

Co-authored-by: Povilas Versockas <povilas.versockas@coralogix.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready to merge Code review completed; ready to merge by maintainers receiver/statsd statsd related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants