Skip to content

[MetricBeat] AWS module support wildcard format dimension value to reduce the metrics count  #18050

Closed

Description

1. Describe the enhancement:

AWS CloudWatch metrics have many dimensions, for example, In AWS/NetworkELB namespace, UnHealthyHostCount has the dimensions:

  • TargetGroup, LoadBalancer
  • TargetGroup, LoadBalancer, AvailableZone

When we query CloudWatch metrics with GetMetricData API, there are two metric.

        {
            "Namespace": "AWS/NetworkELB", 
            "Dimensions": [
                {
                    "Name": "TargetGroup", 
                    "Value": "targetgroup/some-name/222222"
                }, 
                {
                    "Name": "AvailabilityZone", 
                    "Value": "ap-northeast-1a"
                }, 
                {
                    "Name": "LoadBalancer", 
                    "Value": "net/some-lb-name/1111111"
                }
            ], 
            "MetricName": "UnHealthyHostCount"
        },

                {
            "Namespace": "AWS/NetworkELB", 
            "Dimensions": [
                {
                    "Name": "TargetGroup", 
                    "Value": "targetgroup/some-name/222222"
                }, 
                {
                    "Name": "LoadBalancer", 
                    "Value": "net/some-lb-name/1111111"
                }
            ], 
            "MetricName": "UnHealthyHostCount"
        }, 

If metricbeat could limit dimension names scope, after compared to ListMetrics API result, just one metric is retrived from AWS. so this can reduce AWS GetMetircData API cost and metricbeat events amount.

- module: aws
  period: 300s
  metricsets:
    - cloudwatch
  metrics:
    - namespace: AWS/NetworkELB
      statistic: ["Maximum"]
      name: ["UnHealthyHostCount"]
     # demo param
      dimension_names: ["LoadBalancer", "TargetGroup"]
      tags.resource_type_filter: elasticloadbalancing

metricbeat will only make one metric query:

        {
            "Namespace": "AWS/NetworkELB", 
            "Dimensions": [
                {
                    "Name": "TargetGroup", 
                    "Value": "targetgroup/some-name/222222"
                }, 
                {
                    "Name": "LoadBalancer", 
                    "Value": "net/some-lb-name/1111111"
                }
            ], 
            "MetricName": "UnHealthyHostCount"
        }, 

2. Describe a specific use case for the enhancement or feature:

  • reduce AWS GetMetricData cost
  • reduce metricbeat events amount

3. reference

  • aws doc:

https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-cloudwatch-metrics.html

  • inspired by:

https://github.com/prometheus/cloudwatch_exporter#cost

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions