[Metricbeat] Add ability to specify one or more AWS regions in aws module

[kaiyan-sheng]

It will be good for users if they can choose what are the regions they want to collect cloudwatch metrics from. For example, if an user knows they only have EC2 instances on us-west-1 and us-west-2, then there is no need to loop through all regions to query cloudwatch metrics for all instances.

This way, it will make the aws module runs faster and also saves money on unnecessary aws api calls.

[exekias]

+1 Definitely something we should have. The way I imagine this is a regions config key at the module level that affects all metricsets. WDYT? Example:

- module: aws
  metricsets:
    - ec2
  regions: ["us-west-1", "us-west-2"]

[kaiyan-sheng]

@exekias Yes it will affect all metricsets and we can use this regions to replace the current default_region. If users want to apply different regions list to different metricsets, they can do something like this:

- module: aws
  period: 300s
  metricsets:
    - ec2
  access_key_id: '${AWS_ACCESS_KEY_ID:""}'
  secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}'
  session_token: '${AWS_SESSION_TOKEN:""}'
  regions: ["us-west-1", "us-west-2"]
- module: aws
  period: 300s
  metricsets:
    - sqs
  access_key_id: '${AWS_ACCESS_KEY_ID:""}'
  secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}'
  session_token: '${AWS_SESSION_TOKEN:""}'
  regions: ["us-east-1", "us-east-2"]

[exekias]

👍 that should also affect all requested metrics under the cloudwatch metricset

[kaiyan-sheng]

👍 that should also affect all requested metrics under the cloudwatch metricset

@exekias Hmmm but currently in the cloudwatch PR, I have a region parameter for cloudwatch_metrics: https://github.com/elastic/beats/pull/11798/files#diff-a2324399cc5872d37ae79a60f315fee8R144 It defines which region to query for the specific instance. If we add regions on the top level for all metricsets, then region under cloudwatch_metrics should be able to overwrite the top level one?

If we get rid of the region under cloudwatch_metrics, then for cloudwatch the config will look like:

- module: aws
  period: 300s
  metricsets:
    - cloudwatch
  access_key_id: '${AWS_ACCESS_KEY_ID:""}'
  secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}'
  session_token: '${AWS_SESSION_TOKEN:""}'
  regions: ["us-east-1", "us-east-2"]
  cloudwatch_metrics:
    - namespace: AWS/EC2
      metricname: CPUUtilization
      dimensions:
        - name: InstanceId
          value: i-0686946e22cf9494a
    - namespace: AWS/EBS

This means we will query cloudwatch AWS/EBS for us-east-1 and us-east-2 and then query AWS/EC2 instance i-0686946e22cf9494a from us-east-1 and us-east-2. So the user has to make sure if they want to specify one specific instance, then the region it belongs to should be in the regions list.

[kaiyan-sheng]

Just had a chat with @exekias, we decide to use the same field across all metricsets under aws module. This means for cloudwatch metricset, if we want to collect metrics from instances in different regions, we just need to separate them into different config entries:

- module: aws
  period: 300s
  metricsets:
    - cloudwatch
  access_key_id: '${AWS_ACCESS_KEY_ID:""}'
  secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}'
  session_token: '${AWS_SESSION_TOKEN:""}'
  regions: ["us-east-1"]
  cloudwatch_metrics:
    - namespace: AWS/EC2
      metricname: CPUUtilization
      dimensions:
        - name: InstanceId
          value: i-123
    - namespace: AWS/EBS
- module: aws
  period: 300s
  metricsets:
    - cloudwatch
  access_key_id: '${AWS_ACCESS_KEY_ID:""}'
  secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}'
  session_token: '${AWS_SESSION_TOKEN:""}'
  regions: ["us-west-1"]
  cloudwatch_metrics:
    - namespace: AWS/EC2
      metricname: CPUUtilization
      dimensions:
        - name: InstanceId
          value: i-456
    - namespace: AWS/RDS